Health & wellness

Daily Dose

How much can you trust dramatic study findings?

An analysis published last week in the Journal of the American Medical Association examined more than 228,000 clinical trials and found that about 9 percent of them had “very large” findings — showing that a particular treatment was five times or 500 percent more likely to help someone or hurt them compared with a placebo.

These were mostly from small studies with very few measured outcomes such as heart attacks or deaths. If 10 heart attacks occurred in one group and five heart attacks in another, for example, the researchers would have found a five-fold difference between the two groups. But the laws of statistics dictate that smaller samples have larger margins of error.


What’s more, these small studies with dramatic findings were often the first one s of their kind to be published, and repeat studies typically found more modest effects. And almost none of the studies found large differences when comparing the number of deaths between two groups — that outcome is considered the most important measure of how well a medical treatment works or how much something harms.

Bottom line: “Genuine large effects with extensive support from substantial evidence appear to be rare in medicine,” wrote the study authors. That means you should take any study showing something for the first time with a grain of salt — especially those that have dramatic findings. D.K.

Loading comments...
Real journalists. Real journalism. Subscribe to The Boston Globe today.
We hope you've enjoyed your free articles.
Continue reading by subscribing to for just 99¢.
 Already a member? Log in Home
Subscriber Log In

We hope you've enjoyed your 5 free articles'

Stay informed with unlimited access to Boston’s trusted news source.

  • High-quality journalism from the region’s largest newsroom
  • Convenient access across all of your devices
  • Today’s Headlines daily newsletter
  • Subscriber-only access to exclusive offers, events, contests, eBooks, and more
  • Less than 25¢ a week
Marketing image of
Marketing image of