Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics

Abstract

We present the first study to investigate Large Language Models (LLMs) in answering radiation oncology physics questions. Because popular exams like AP Physics, LSAT, and GRE have large test-taker populations and ample test preparation resources in circulation, they may not allow for accurately assessing the true potential of LLMs. This paper proposes evaluating LLMs on a highly-specialized topic, radiation oncology physics, which may be more pertinent to scientific and medical communities in addition to being a valuable benchmark of LLMs. We developed an exam consisting of 100 radiation oncology physics questions based on our expertise at Mayo Clinic. Four LLMs, ChatGPT (GPT-3.5), ChatGPT (GPT-4), Bard (LaMDA), and BLOOMZ, were evaluated against medical physicists and non-experts. ChatGPT (GPT-4) outperformed all other LLMs as well as medical physicists, on average. The performance of ChatGPT (GPT-4) was further improved when prompted to explain first, then answer. ChatGPT (GPT-3.5 and GPT-4) showed a high level of consistency in its answer choices across a number of trials, whether correct or incorrect, a characteristic that was not observed in the human test groups. In evaluating ChatGPTs (GPT-4) deductive reasoning ability using a novel approach (substituting the correct answer with "None of the above choices is the correct answer."), ChatGPT (GPT-4) demonstrated surprising accuracy, suggesting the potential presence of an emergent ability. Finally, although ChatGPT (GPT-4) performed well overall, its intrinsic properties did not allow for further improvement when scoring based on a majority vote across trials. In contrast, a team of medical physicists were able to greatly outperform ChatGPT (GPT-4) using a majority vote. This study suggests a great potential for LLMs to work alongside radiation oncology experts as highly knowledgeable assistants.

References in corpus (10)

• Training language models to follow instructions with human feedback
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
• LaMDA: Language Models for Dialog Applications
• Capabilities of GPT-4 on Medical Challenge Problems
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Improving alignment of dialogue agents via targeted human judgements
• DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4
• ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models
• Mask-guided BERT for Few Shot Text Classification

Cited by in corpus (8)

• Differentiate ChatGPT-generated and Human-written Medical Texts
• Artificial General Intelligence for Medical Imaging Analysis
• Evaluating Search Engines and Large Language Models for Answering Health Questions
• Medical Misinformation in AI-Assisted Self-Diagnosis: Development of a Method (EvalPrompt) for Analyzing Large Language Models
• Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment
• A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options
• FLoRA: An Advanced AI-Powered Engine to Facilitate Hybrid Human-AI Regulated Learning
• Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications