The study "Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum," published in JAMA Internal Medicine, triggered a storm that their authors didn't expect. The media reported that ChatGPT is "more accurate, detailed, empathetic or even better than doctors.
The study compared physician and chatbot responses to patient questions asked in a public social media forum – a team of healthcare professionals rated 195 randomly selected queries. Outcomes: The chatbot's responses were preferred to those of the physician and rated significantly higher in quality and empathy.
In 78.6%, the chatbot's responses were considered to be better than the doctor's. More specifically, the doctors' answers were significantly shorter than the chatbot's answers (52 words vs. 211 words). Besides, they were rated as significantly higher quality than the doctors' answers. In addition, the chatbot's answers were assessed as significantly more empathetic than the doctors'. The difference was huge: the frequency of compassionate or very empathetic responses for the chatbot was 9.8 times higher than for doctors.
The authors concluded that generative AI has the potential to generate answers to standard patient questions. In the future, doctors could edit them instead of writing them from scratch. And that means significant improvements, especially in repetitive work - AI could even reduce doctors' job burnout and improve patient outcomes.
Why is it ok that ChatGPT outperformed doctors?
AI will not replace doctors; the new study gives no reason to think otherwise. But it has shown that AI will almost certainly help doctors in their work, answering patients' questions more empathetically. Patients demand empathy, but this feature of the patient-doctor relationship is often overworked for a simple reason: busy doctors don't have time to prioritize good manners. Instead, they have to prioritize patient outcomes and safety. AI can fill this gap. Especially as health care is rapidly moving to online channels – telemedicine services, now also provided via chat, are on the rise. With acute doctor shortages in all countries, AI could be a massive help since typing responses is time-consuming and frustrating. In fact, it's another kind of paperwork that distract healthcare professionals from direct contact with patients.9.8 times more empathetic responses than physiciansBy augmenting the clinical workflows with AI, both parties should profit – the doctor will gain a valuable supporter, while the patient will - high-quality answers written in an understandable style and clinically verified by medical professionals, tailored to the patient's needs (including for example, age or mental health). Of course, it is also crucial that AI is not only empathetic but also reliable. But we are also seeing a lot of progress in this area: In early 2023, the new Large Language Model (LLM) Med-PaLM 2 passed the medical exam by answering more than 80% of the questions correctly, twice as many as the score achieved at the beginning of 2022. AI is almost ready to augment doctors' competencies in practical work, especially since yet another study shows that ChatGPT can even be used to generate clinical letters. However, despite the snowballing number of scientific studies on ChatGPT, the barrier may be the issue of liability. Even if doctors were only to check the accuracy of ChatGPT responses, they are still responsible for the final diagnosis or recommendations forwarded to the patient. As long as ChatGPT is not a medical device, the revolution won't start. We must also consider that editing AI-generated texts can be as time-consuming and depressing as writing them from scratch.
