Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Study

Abstract

OpenNotes gives patients access to their EHR notes, but dense medical jargon limits comprehension. We evaluate closed-source and open-source LLMs for extracting and prioritizing the jargon terms most relevant to individual patients, using 90 expert-annotated EHR notes. We test combinations of general vs. structured prompts, zero-shot vs. few-shot prompting, fine-tuning, and GPT-4o-based data augmentation, the last paired with a ranking technique to refine training in low-resource settings. To assess the effect of dataset size, we fine-tune on augmented datasets scaled from 10 to 9,995 examples. All settings are evaluated with 10-fold cross-validation, reporting F1 and Mean Reciprocal Rank (MRR) under two string-matching criteria (relaxed matching and Jaccard Index), followed by an error analysis of model outputs. Open-source models performed best when fine-tuned on the gold-standard dataset: under Jaccard-based matching, DeepSeek 8B achieved the top F1 (0.431, SD 0.046) and BioMistral 7B the top MRR (0.577, SD 0.109). Under relaxed matching, however, open-source models did not match closed-source performance even with augmentation or fine-tuning. Few-shot prompting offered no advantage over zero-shot in vanilla models; prompting style substantially affected results; fine-tuning on a small gold-standard set improved performance; and data augmentation matched or exceeded fine-tuning, though its benefit depended heavily on augmented-data quality. These findings show that prompting strategy, fine-tuning, and data augmentation each meaningfully improve LLM performance on patient-centered jargon extraction in low-resource clinical settings.

28pages, 5 figures, 6 tables

References in corpus (12)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
• Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
• What Makes Good In-Context Examples for GPT-$3$?
• Zero-shot information extraction from radiological reports using ChatGPT
• Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data
• Unsupervised Ensemble Ranking of Terms in Electronic Health Record Notes Based on Their Importance to Patients
• LLMs in Biomedicine: A study on clinical Named Entity Recognition
• Real-Fake: Effective Training Data Synthesis Through Distribution Matching
• Evaluating Language Models as Synthetic Data Generators
• How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes