Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization

Abstract

Analyzing vast textual data and summarizing key information from electronic health records imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown promise in natural language processing (NLP), their effectiveness on a diverse range of clinical summarization tasks remains unproven. In this study, we apply adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks: radiology reports, patient questions, progress notes, and doctor-patient dialogue. Quantitative assessments with syntactic, semantic, and conceptual NLP metrics reveal trade-offs between models and adaptation methods. A clinical reader study with ten physicians evaluates summary completeness, correctness, and conciseness; in a majority of cases, summaries from our best adapted LLMs are either equivalent (45%) or superior (36%) compared to summaries from medical experts. The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information. Our research provides evidence of LLMs outperforming medical experts in clinical text summarization across multiple tasks. This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.

27 pages, 19 figures

References in corpus (25)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• A Survey of Large Language Models
• Scaling Instruction-Finetuned Language Models
• Exploring the Limits of Language Modeling
• QLoRA: Efficient Finetuning of Quantized LLMs
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Prompt Engineering for Healthcare: Methodologies and Applications
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
• MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data
• UL2: Unifying Language Learning Paradigms
• Hyena Hierarchy: Towards Larger Convolutional Language Models
• Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
• LongNet: Scaling Transformers to 1,000,000,000 Tokens
• Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
• Towards Generalist Biomedical AI
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
• Toward expanding the scope of radiology report summarization to multiple anatomies and modalities
• Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibration
• RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models
• SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization
• Design Considerations for High Impact, Automated Echocardiogram Analysis

Cited by in corpus (23)

• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• Delving into LLM-assisted writing in biomedical publications through excess vocabulary
• Foundation Models in Radiology: What, How, When, Why and Why Not
• Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation
• HANNA: Hard-constraint Neural Network for Consistent Activity Coefficient Prediction
• From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine
• The Foundational Capabilities of Large Language Models in Predicting Postoperative Risks Using Clinical Notes
• Best Practices for Large Language Models in Radiology
• Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset
• The current status of large language models in summarizing radiology report impressions
• A dataset and benchmark for hospital course summarization with adapted large language models
• Aligning Large Language Models for Enhancing Psychiatric Interviews Through Symptom Delineation and Summarization: Pilot Study
• Leveraging Open-Source Large Language Models for Clinical Information Extraction in Resource-Constrained Settings
• Enhancing Abstractive Summarization of Scientific Papers Using Structure Information
• A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination
• Transforming Agency. On the mode of existence of Large Language Models
• ADAM: An AI Reasoning and Bioinformatics Model for Alzheimer's Disease Detection and Microbiome-Clinical Data Integration
• Enhancing Hepatopathy Clinical Trial Efficiency: A Secure, Large Language Model-Powered Pre-Screening Pipeline
• LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models
• Align Generative Artificial Intelligence with Human Preferences: A Novel Large Language Model Fine-Tuning Method for Online Review Management
• Evaluation of the phi-3-mini SLM for identification of texts related to medicine, health, and sports injuries
• Performance Assessment Strategies for Language Model Applications in Healthcare
• Medical Incident Causal Factors and Preventive Measures Generation Using Tag-based Example Selection in Few-shot Learning