In a research letter by David Chen, Rod Parsa, Andrew Hope and colleagues, published in JAMA Oncology, the responses of artificial intelligence (AI) chatbots to cancer-related questions from social media were compared with those of licensed physicians. The study assessed the empathy, quality and readability of the responses.
The equivalence trial involved six oncology physician evaluators who rated the responses to 200 patient questions from a public online forum, Reddit’s r/AskDocs, collected between January 1, 2018, and May 31, 2023. Three AI chatbots—GPT-3.5, GPT-4, and Claude AI—were tested against the physicians’ responses.
PODCAST EXCLUSIVE TO THE ONCOLOGY NETWORK | LISTEN NOW
The evaluation focused on three primary outcomes: the quality, empathy and readability of the responses, each rated on a Likert scale from 1 (very poor) to 5 (very good). Additionally, the readability of the responses was assessed using the Flesch-Kincaid Grade Level.
Claude AI Outperforms Clinicians
The findings revealed that the best-performing AI chatbot (Claude AI) consistently outperformed physicians in all three metrics. Chatbot responses were rated higher in quality (mean score 3.56 vs. 3.00, P < .001), empathy (mean score 3.62 vs. 2.43, P < .001), and readability (mean score 3.79 vs. 3.07, P < .001). However, the readability level of physician responses (mean Flesch-Kincaid Grade Level 10.11) was comparable to Claude AI (mean 10.31) but lower than GPT-3.5 (mean 12.33) and GPT-4 (mean 11.32).
AI Could Reduce Onco-Burnout
These results indicate that AI chatbots can generate high-quality, empathetic and readable responses to cancer-related patient inquiries. The study suggests that future development could see AI chatbots and physicians collaborating in clinical practice to enhance patient care and reduce physician burnout. Chatbots could provide initial empathetic response templates, which physicians can then refine for medical accuracy using their expertise. Further research is needed to explore the implementation and effects of chatbot-assisted patient interactions.
