Eye Doctors Find That WebMD Symptom Checker Was Wrong More Than Half The Time

Symptom checkers didn't always catch cases that needed emergency care.






In 2015, The BMJ (formerly British Medical Journal) did a thorough audit of online symptom checkers. It found that, on average, the sites listed the correct diagnosis first only about a third of the time. Carl Shen, an ophthalmology resident at McMaster University, has led a team of researchers in a small-scale follow-up looking specifically at eye health and got equally concerning results: the correct diagnoses popped up first only a quarter of the time.

The results, presented at the Annual Meeting of the American Academy of Ophthalmology this week, are early and provisional, but Dr. Shen and his team are planning a larger follow-up study. In the meantime, WebMD has done an update of its algorithm.

Vignettes of unpleasantries

To assess WebMD’s accuracy, Shen and his colleagues compiled 42 eye-health “clinical vignettes” based on the medical literature. A decidedly unpleasant vignette of someone suffering from acute angle-closure glaucoma, for instance, describes a “44-year-old woman present[ing] to ER... with severe pain around her right eye of four-hour duration... She is also nauseated and has thrown up once... Intraocular pressure is extremely elevated.”

These vignettes were then boiled down the barebones symptoms so that they could be easily entered into the symptom checker twice: once by a medical professional, and once by a non-professional. That helped to account both for how non-specialists would interact with the tool and for how different people might describe the same condition in different ways. Because the diagnoses of the clinical cases were already known, the outcome of the symptom checker could be compared to the known, correct diagnosis.

Out of the 42 cases, the correct diagnosis appeared somewhere in the list 18 times—just less than half. The correct result appeared at the top of the list in only 11 cases. And because the main diagnosis was incorrect most of the time, the recommended course of action was frequently off as well. Out of the 18 vignettes that were based on a clinical emergency, the symptom checker correctly recommended the necessary urgent care in just seven of them.

These results are a bit worse than the findings of the 2015 BMJ investigation, which reported the correct diagnosis appearing in the top 20 symptom-checker options 58 percent of the time and appropriate recommendations for emergency care given in 80 percent of the medical emergency cases. But Dr. Shen’s ophthalmology study is also much smaller than the BMJ’s, which looked at 23 different symptom checkers and compiled results from many different people entering the same symptoms. So the ophthalmology study might be less reliable. Shen and his team are also much earlier in the research process than the BMJ investigation, and they are considering crowdsourcing as a way to capture how different people would describe the same condition differently.

Software updates

It’s also important to note that in the time since both of these studies were conducted, WebMD has updated its algorithm, partly as a result of the BMJ research. Where the old symptom checker required users to start by clicking on a body part, the new version just needs a description of the symptoms. “After testing the new version of the WebMD Symptom Checker, our analysis of 77 varied cases showed an accuracy rate of 70 percent for the condition appearing in the top 3,” says WebMD Medical Director Michael Smith.

Independent assessment of that success rate and an exploration of whether it looks the same across different areas of health could follow in future research. As it stands, the picture might be a bit sunnier than Shen and his colleagues suggest, but it’s clear that online symptom checkers shouldn’t be used in isolation to make medical decisions. They’re “intended for informational purposes only,” says Dr. Smith, emphasizing that physicians should “engage their patients in conversation to help them understand how best to use the information.”

Helping doctors to have that conversation from an informed starting point is precisely why Shen was interested in the question. With more patients arriving at the doctor’s office having researched their symptoms themselves, he and his colleagues write in their conference presentation, “it is important for ophthalmologists to be aware of the capabilities and limitations of available health information tools.”

Source