AI-based chatbots can be a useful tool for medical students, albeit with significant limitations. The main benefits lie in helping with exam preparation and practice, according to research from the University of Western Ontario in using ChatGPT in diagnosis, published in the scientific journal Plos One. AI applications based on deep and machine learning have been used for years in medical diagnostics.
The reason for the study was that previous research showed how ChatGPT was able to answer biomedical and clinical questions on the United States Medical Licensing Examination (USMLE) accurately, at a level that approached or exceeded the threshold. This study also showed that ChatGPT's accuracy was characterized by a high density of insights, indicating its potential to generate new insights and aid in medical education.
While these results have sparked discussions about possible implications for ChatGPT in health care, they also highlight the potential use of this tool in medical education, according to researchers at the University of Western Ontario. Although ChatGPT's ability to answer concise, encyclopedic questions has been explored, the quality of answers to complex medical cases, however, remains unclear.
Poor in diagnostics
In only 49 percent of the 150 medical cases presented, ChatGPT 3.5 came up with a correct answer, the Canadian study found. In three-quarters of all cases, the gene AI application did manage to strip away misdiagnoses, but without being able to pinpoint what did constitute a patient's condition.
ChatGPT in its current form is not accurate as a diagnostic tool, the researchers write. ChatGPT does not necessarily provide factual accuracy, despite the vast amount of information it has been trained on. Based on our qualitative analysis, ChatGPT struggles to interpret laboratory values and imaging results and may overlook important information relevant to diagnosis.'
Good as an educational tool
ChatGPT is useful as an educational tool, however, the scientists continue. ChatGPT was generally right in ruling out a specific diagnosis and providing reasonable next diagnostic steps. Moreover, the answers were easy to understand, which could be a potential advantage in simplifying complex concepts for medical students.
'Our results should guide future research on harnessing the potential educational benefits of ChatGPT, such as simplifying medical concepts and providing guidance on differential diagnoses and next steps.'
Solid research
According to Roger Rennenberg, internist and training director of medicine at Maastricht UMC+, the Canadian scientists' research is sound. For example, the variety of cases - from the period 2021-2023 - is large, he says in the Volkskrant. However, he does regret that ChatGPT 3.5 was used, since ChatGPT 4 is now also available. This version is trained with data from the period from which the cases originate.
Rennenberg argues that ChatGPT as an LMM (large language model) predicts the most logical word after the previous one. It has no real knowledge about the human body. Moreover, it cannot, as a physician must, read between the lines. But since ChatGPT is good at drafting examination and treatment plans and produces mostly comprehensible texts, medical students can use the model to simplify or summarize complicated texts.
Last year, a study published in the journal Jama Internal Medicine found that ChatGPT answered online medical questions better and more compassionately than doctors. In the long run, therefore, an LLM application such as ChatGPT could certainly be a gamechanger, but researchers are cautious about using this capability in clinical settings.
