Metro

Looking to diagnose your ailments online? Be careful

Study finds online ‘symptom checkers’ often get it wrong

A Harvard review of online “symptom checkers” found wide variation in accuracy and a high rate of wrong answers.
istockphoto
A Harvard review of online “symptom checkers” found wide variation in accuracy and a high rate of wrong answers.

People who use online tools to diagnose their ailments should proceed with caution — because these “symptom checkers” often get it wrong, according to Harvard Medical School researchers.

Their study, published this week in the medical journal BMJ, provides the first assessment of a burgeoning phenomenon: online programs that crunch patients’ descriptions of what ails them and spit out a list of possible diagnoses, often with advice on how to respond.

Millions of people turn to such services. They go by names such as iTriage and FreeMD, or are offered on well-known health websites run by WebMD or the Mayo Clinic. Some are hosted by medical schools, hospital systems, insurance companies, or government agencies. A few have been around five or more years, but many are new, reflecting the growing use of the Internet to seek health information.

Advertisement

The Harvard review found wide variation in accuracy and a high rate of wrong answers. But overall, the accuracy was comparable to that of telephone triage lines that primary care practices use to field calls from ill patients.

Get Metro Headlines in your inbox:
The 10 top local news stories from metro Boston and around New England delivered daily.
Thank you for signing up! Sign up for more newsletters here

“Even though their performance is not great, they do provide some information,” said Dr. Ateev Mehrotra, a Harvard health care policy professor and study author. “But don’t take this as gospel.”

To test the online symptom checkers, researchers invented 45 patients, each with a set of symptoms clearly indicating a specific illness. A third of these imaginary patients had conditions that warranted immediate medical attention. A third had ailments requiring a doctor’s care, but not urgently, and the remaining conditions didn’t need treatment.

The researchers entered their made-up patients’ symptoms into 23 symptom checkers, and got back from each a list of possible diagnoses. Overall, the results broke down this way:

 34 percent of symptom checkers listed the correct diagnosis first.

Advertisement

 51 percent included the right answer in the top three possible diagnoses.

 58 percent placed the correct diagnosis somewhere within the top 20.

The range among the various services was wide, however. A symptom checker named DocResponse listed the correct diagnosis first in 50 percent of inquiries, while MEDoctor achieved that only 5 percent of the time.

Fifteen of the 23 symptom checkers also ventured into the tricky task of advising people on what to do after getting the diagnosis. For those whose symptoms suggested an urgent need for care, the checker got it right 80 percent of the time. For those who needed care, but not urgently, the response was correct 55 percent of the time.

But when it came to those conditions people can manage on their own, the software played it safe: Two-thirds were advised to get medical attention despite not needing it.

Advertisement

Despite their limitations, Mehrotra said, symptom trackers are preferable to asking no questions — or posing questions to search engines such as Google, which draw from throughout the Internet and can lead a patient down a rabbit hole of misinformation.

Dean Stephens, chief executive of Healthline, a health-information website whose symptom tracker gets 6.5 million visits per month, pointed out that doctors usually see a patient before making a diagnosis, and even they aren’t always right. “You shouldn’t expect a high diagnosis rate with these symptom trackers given the limited available information that you’re receiving from an anonymous, invisible user,” he said.

A key question, which the study doesn’t address, is how people respond once they get the computer-generated diagnosis.

Susannah Fox, chief technology officer at the US Department of Health and Human Services and a former researcher at the Pew Research Center, said her work at Pew found that one-third of American adults go online to diagnose their illnesses. But people rarely base their health decisions exclusively on what they find there, she said.

A majority also turn to medical professionals for information and advice, followed by friends and family, and then others who have the same condition.

“The reality is, people are using these tools and consulting Dr. Google whether we like it or not,” Fox said. “What this study shows is that we have an opportunity to open up a new avenue to reach people with quality information.”

Dr. John Wilkinson, a family medicine physician at the Mayo Clinic in Rochester, Minn., and an editor of the Mayo Clinic Symptom Checker, said that when editing information about a symptom, he always struggles to find an “appropriate tone, so that people are not unduly alarmed, but neither are they inappropriately reassured. And finding that tone is difficult.”

Dave deBronkart, an activist for patient empowerment best known as “e-Patient Dave,” praised the study for showing consumers and doctors that “some checkers are pretty darn good — much better than nothing” and “some are flat-out junk.” He recommended that patients try two or three symptom checkers, rather than assuming any one is correct.

But he welcomed symptom checkers into the patient arsenal. “All in all, I view this as another step in the direction of more and better information in consumer hands,” he wrote in an e-mail, cautioning that “online info is just a first guess . . . not something to stake your life on.”

Mehrotra surmised that the symptom checkers’ popularity may result, in part, from the prevalence of high-deductible health plans, which require consumers to pay thousands of dollars out of pocket before their insurance kicks in. “That’s one thing that’s making people think twice about going to the doctor,” he said. “Patients are left with this struggle: Do I need care and where the heck do I go?”

Dr. Michael Smith, chief medical editor of WebMD, defended his website’s symptom checker, which gets about 4 million visits a month, and described it as accurate and helpful.

“People want information, and they want information now. WebMD is readily accessible to them,” he said. “It’s an engaging way for people to get to additional information about their symptoms.”

Felice J. Freyer can be reached at felice.freyer@globe.com. Follow her on Twitter @felicejfreyer