10 November 2016

Online Symptom Checkers List Correct Diagnosis First Only One-Third of the Time

Patients increasingly rely on online sources of medical information, including WebMD and Google. Patients’ increasing engagement in their healthcare is a good thing, but they often come in with an incorrect diagnosis based on online research, as Dr. Catherine Polera, Chief Medical Officer for Sheridan’s Emergency Services division explained in a MedPage Today article. Now physicians can show their patients concrete evidence that a doctor’s diagnosis is far more likely to be correct than that of a computer algorithm.

Researchers at Harvard Medical School and Brigham and Women’s Hospital in Boston recently tested the diagnostic accuracy of trained physicians compared to that of online symptom checkers. In a previous study published last July in BMJ, the researchers had evaluated the diagnostic accuracy of 23 symptom checkers (both websites and apps) using 45 clinical vignettes. According to a research letter published in JAMA Internal Medicine, the new study asked 234 internal medicine, family practice and pediatrics physicians to evaluate the same vignettes, each of which was solved by at least 20 physicians. As with the symptom checkers in the previous study, the physicians were asked to identify the most likely diagnosis along with two additional possible diagnoses for each case. The vignettes comprised 15 high-acuity, 15 medium-acuity and 15 low-acuity condition scenarios involving 26 common and 19 uncommon conditions, and included the symptoms and history of the patient but no physical exam or test findings. 

Physicians Outperformed Symptom Checkers in Diagnostic Accuracy by a 2-to-1 Margin

The physicians significantly outperformed the symptom checkers in all scenarios. Across all cases, the physicians listed the correct diagnosis first 72.1% of the time, compared to 34% of the time for the symptom checkers. Physicians also included the correct diagnosis in their top three diagnoses 84.3% of the time, compared to 51.2% of the time for the symptom checkers.

Interestingly, physicians were more likely to list the correct diagnosis first for high-acuity cases (71% of the time) than for low-acuity cases (65.3%), and for uncommon cases (75.5%) versus common cases (69.6%). Conversely, symptom checkers were more likely to list the correct diagnosis first for low-acuity cases (40.5%) than for high-acuity cases (24.3%), and for more common cases (38.1%) versus uncommon cases (28.1%). 

Can Technology Help Improve Diagnostic Accuracy?

Despite their much higher diagnostic accuracy rate versus the algorithms, physicians provided an incorrect diagnosis in about 15% of the cases. Per a recent Modern Healthcare article, Ateev Mehrotra, M.D., author of the new study and associate professor of health care policy and medicine at Harvard Medical School, said it would be useful to study how computers could improve physicians' diagnostic accuracy. In a MedPage Today article responding to the study’s findings, Dr. Art Papier, associate professor of dermatology and medical informatics at the University of Rochester School of Medicine and Dentistry and co-founder and CEO of VisualDx, said new professional systems that model variation in presentation already exist and “clinicians should be augmenting their knowledge with tools to help them do their job better.” And the new Watson Medical Imaging Collaborative is using IBM’s Watson supercomputer to try to help physicians reduce diagnostic errors.