Next time you hear about a new "study"....

[alex2678]

A co-worker of mine showed me an article, which I had read last year, that made the claim "Vitamin E may be deadly." It mentioned that those who took Vitamin E supplements were more likely to die sooner than those who did not. (http://www.seniorjournal.com/NEWS/Nutrition-Vitamins/4-11-10VitaminE.htm) Complete bull in my opinion. Since there seems to be a "new" story or theory on health or nutrition everyday either on the new or in the paper, I thought it might me a good idea to keep this in mind:


Most scientific papers are probably wrong

30 August 2005
Kurt Kleiner


Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.

John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.

"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.

In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.
Massaged conclusions

Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.

Odds get even worse for studies that are too small, studies that find small effects (for example, a drug that works for only 10% of patients), or studies where the protocol and endpoints are poorly defined, allowing researchers to massage their conclusions after the fact.

Surprisingly, Ioannidis says another predictor of false findings is if a field is "hot", with many teams feeling pressure to beat the others to statistically significant findings.

But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research.

"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.

Journal reference: Public Library of Science Medicine (DOI: 10.1371/journal.pmed.0020124)

 

[*Bunny*]

Massaged conclusions

I like that

"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," ... Sure is

 

[GSP]

Good Stuff. The problem isn't so much the research itself, as the media outlets taking every new study as a new breakthrough or a definitive answer. When in fact, a study is simply one more data point in a much more complex analysis.

I could rattle off a dozen studies that have been taken out of context in the last year alone by the media and/or special interest groups (special interest groups spinning data is a huge part of the problem as well).

 

[fortunatesun]

Something else to consider is who is funding the study. Even academic institutions are very aware of where their research grants come from. Hypotheses are often constructed in a way that don't agree with best scientific method.

 

[Lifterforlife]

Something else to consider is who is funding the study. Even academic institutions are very aware of where their research grants come from. Hypotheses are often constructed in a way that don't agree with best scientific method.

Great post. This is indeed true. Follow the money.

Many studies are so poorly done. And then also, any study is only as good as the researchers doing the research, and their interpretation of the data. I know someone who talks with some of these folks(written books) who do these protein studies, etc., and the head researcher will many times tell how they even know the study is flawed because of this or this reason.

If you go into a study with a preconcieved notion also, it is hard to be objective.

 

[javaguru]

Remember the miraculous discovery of cold fusion years ago reported by all the news services? Well, nobody could replicate the experiment and it turned out to be a complete fabrication. Did they actually think they would get away with fabricating the discovery of cold fusion??? BTW, HMB feels like deca!

When it comes to human health and nutrition there are SO many variables and controlling the test and control groups is practically impossible. Look at prescription drugs, they have a long peer reviewed process to ensure safety and drugs get pulled left and right for previously unsuspected serious health concerns.