Capabilities of GPT-4 on Medical Challenge Problems

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.

arXiv comment35 pages, 15 figures; added GPT-4-base model results and discussion

Cited by in corpus (23)

• ChatGPT: Jack of all trades, master of none
• Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
• Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
• Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices
• 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
• How understanding large language models can inform the use of ChatGPT in physics education
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• LLM for SoC Security: A Paradigm Shift
• Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation
• How Good is ChatGPT at Face Biometrics? A First Look into Recognition, Soft Biometrics, and Explainability
• MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
• Leveraging Large Language Models for Patient Engagement: The Power of Conversational AI in Digital Health
• From Text to Image: Exploring GPT-4Vision's Potential in Advanced Radiological Analysis across Subspecialties
• Large-Scale Text Analysis Using Generative Language Models: A Case Study in Discovering Public Value Expressions in AI Patents
• Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data
• Evaluation of GPT and BERT-based models on identifying protein-protein interactions in biomedical text
• Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation
• The Last JITAI? Exploring Large Language Models for Issuing Just-in-Time Adaptive Interventions: Fostering Physical Activity in a Conceptual Cardiac Rehabilitation Setting
• Streamlining evidence based clinical recommendations with large language models
• Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming
• BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment