OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models

Abstract

LLMs have become increasingly capable at accomplishing a range of specialized-tasks and can be utilized to expand equitable access to medical knowledge. Most medical LLMs have involved extensive fine-tuning, leveraging specialized medical data and significant, thus costly, amounts of computational power. Many of the top performing LLMs are proprietary and their access is limited to very few research groups. However, open-source (OS) models represent a key area of growth for medical LLMs due to significant improvements in performance and an inherent ability to provide the transparency and compliance required in healthcare. We present OpenMedLM, a prompting platform which delivers state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks. We evaluated a range of OS foundation LLMs (7B-70B) on four medical benchmarks (MedQA, MedMCQA, PubMedQA, MMLU medical-subset). We employed a series of prompting strategies, including zero-shot, few-shot, chain-of-thought (random selection and kNN selection), and ensemble/self-consistency voting. We found that OpenMedLM delivers OS SOTA results on three common medical LLM benchmarks, surpassing the previous best performing OS models that leveraged computationally costly extensive fine-tuning. The model delivers a 72.6% accuracy on the MedQA benchmark, outperforming the previous SOTA by 2.4%, and achieves 81.7% accuracy on the MMLU medical-subset, establishing itself as the first OS LLM to surpass 80% accuracy on this benchmark. Our results highlight medical-specific emergent properties in OS LLMs which have not yet been documented to date elsewhere, and showcase the benefits of further leveraging prompt engineering to improve the performance of accessible LLMs for medical applications.

References in corpus (12)

• PaLM: Scaling Language Modeling with Pathways
• Emergent Abilities of Large Language Models
• Gemini: A Family of Highly Capable Multimodal Models
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Code Llama: Open Foundation Models for Code
• Measuring Massive Multitask Language Understanding
• Large Language Models Encode Clinical Knowledge
• Mixtral of Experts
• Artificial Intelligence for Health Message Generation: Theory, Method, and an Empirical Study Using Prompt Engineering
• What learning algorithm is in-context learning? Investigations with linear models
• What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams

Cited by in corpus (2)

• From Intention To Implementation: Automating Biomedical Research via LLMs
• Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond