An analysis published last week in the Journal of the American Medical Association examined more than 228,000 clinical trials and found that about 9 percent of them had “very large” findings — showing that a particular treatment was five times or 500 percent more likely to help someone or hurt them compared with a placebo.
These were mostly from small studies with very few measured outcomes such as heart attacks or deaths. If 10 heart attacks occurred in one group and five heart attacks in another, for example, the researchers would have found a five-fold difference between the two groups. But the laws of statistics dictate that smaller samples have larger margins of error.
What’s more, these small studies with dramatic findings were often the first one s of their kind to be published, and repeat studies typically found more modest effects. And almost none of the studies found large differences when comparing the number of deaths between two groups — that outcome is considered the most important measure of how well a medical treatment works or how much something harms.
Bottom line: “Genuine large effects with extensive support from substantial evidence appear to be rare in medicine,” wrote the study authors. That means you should take any study showing something for the first time with a grain of salt — especially those that have dramatic findings.