Document Type
Abstract
Publication Date
2-11-2026
Abstract
Introduction
Large Language Models (LLMs) have rapidly entered consumer-facing health information platforms, yet their reliability for patient education remains uncertain. Otolaryngology involves complex surgical procedures where clear perioperative counseling is essential for patient understanding and safety. This project evaluates whether LLMs can provide accurate and readable answers to common perioperative questions for septoplasty and cochlear implant procedures. We hypothesized that LLMs would generate accurate responses with readability levels appropriate for patients, with ChatGPT performing best.
Methods
Two common pre-operative and two post-operative questions for septoplasty and cochlear implantation were converted into standardized prompts. Each prompt was submitted to ChatGPT-5.1 and Gemini 2.5. Readability was assessed using the Flesch Reading Ease (FRE) metric. Accuracy scoring will be completed by three board-certified otolaryngologists using a 1– 5 Likert scale.
Results
ChatGPT responses averaged an FRE score of 42.5, while Gemini responses averaged 43.8, indicating college-level reading difficulty. These values fall well below federal recommendations for 6th–8th grade readability in patient education. A paired comparison showed no significant difference in readability between ChatGPT and Gemini (p > 0.05). Accuracy data is currently being collected.
Discussion/Conclusions
Preliminary findings suggest that LLM-generated perioperative responses are too complex for patients to comprehend regularly. Neither LLM demonstrated superior readability, and both produced content above recommended literacy standards. These results suggest that ChatGPT and Gemini, in their current versions, are insufficient standalone tools for perioperative patient education. Pending accuracy results will further clarify their potential ability for misinformation or education.
Recommended Citation
Ganju, BS, Shaunak and Luginbuhl, MD, Adam, "Evaluating the Clinical Utility of Large Language Models in Otolaryngology Patient Education" (2026). Phase 1. Paper 2.
https://jdc.jefferson.edu/si_dh_2028_phase1/2
Language
English
Comments
Presented at the 2026 Scholarly Inquiry (SI) Research Project Symposium.