On Second Thought: Flawed review foundations

MELBOURNE, AUSTRALIA - MARCH 06: Rowers take to the water for an early morning training session on the Yarra River, March 6, 2013 in Melbourne, Australia. Rowing has been held on the Yarra River for over 150 years. Seven clubs are based in the historic boat sheds on the Alexandra Gardens, they are Banks Rowing Club, Mercantile Rowing Club ,La Trobe University Rowing Club, Melbourne Rowing Club, Melbourne University Boat Club, Yarra Yarra Rowing Club and the Richmond Rowing Club. Rowers from local private schools, elite athletes and members of the public train on the Yarra River on a daily basis from early morning to midmorning then again in the afternoonon. (Photo by Robert Cianflone/Getty Images)
Robert Cianflone/Getty Images
Melbourne, Australia.

Systematic reviews make some of the best scientific evidence. Reviewing the entire literature on a given issue filters out the glare of individual studies, giving researchers a better sense of the big picture — and giving people in clinical fields, such as nurses, better guidance about how to treat patients. (Britain’s famed Cochrane reviews, for example, look at everything from treatment for back pain to the therapies for hepatitis C infection.) The problem comes when these systematic reviews include faulty studies.

Recently, nursing researchers in Australia were alarmed to find 23 reviews citing studies that had been retracted for being unreliable. Of those, 18 referred to studies that were pulled only after the publication of the reviews. That suggests bad luck, rather than sloppiness, was to blame. But five of the 23 reviews cited previously retracted studies, indicating that the authors had not bothered to check. “Clinicians may continue to base decisions on evidence from systematic reviews that is unsound,” said Richard Gray, of La Trobe University, who led the new Australian study. “This is a concerning — largely unrecognized — threat to patient safety.”