33 citations · 88 across the 8 of their papers we have counts for
9 papers
Scaling Clinical Judgment to Evaluate Medical AI
Thomas A. Buckley, Zahir Kanjee, Peter G. Brodeur +15
Blinded physician evaluation has been considered by many to be the gold standard for assessing clinical reasoning in large language models (LLMs). This is difficult to scale; thus,…
Scalable Oversight for AI in Mental Health: Lessons from 350,000 AI Coaching Conversations between Therapy Sessions
Matthew A. Scult, John L. Havlik, Kevin Ramotar +2
Clinician review of every AI output is often proposed as a safeguard in mental healthcare, but vigilance research suggests this approach fails at scale and may paradoxically reduce…
HealMed: Multilingual Evaluation of Large Language Models in Medicine
Yingjian Chen, Fan Gao, Sherry T. Tong +42
We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn…
Towards Expert-level Medical AI for Real-time Video Consultations
Mahvish Nagda, Jihyeon Lee, Matthew Thompson +37
Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text…
Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates
Liam G. McCoy, Fateme Nateghi Haredasht, Kanav Chopra +15
This study evaluates the capacity of large language models (LLMs) to generate structured clinical consultation templates for electronic consultation. Using 145 expert-crafted templ…
MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
Suhana Bedi, Hejie Cui, Miguel Fuentes +78
While large language models (LLMs) achieve near-perfect scores on medical licensing exams, these evaluations inadequately reflect the complexity and diversity of real-world clinica…
Superhuman performance of a large language model on the reasoning tasks of a physician
Peter G. Brodeur, Thomas A. Buckley, Zahir Kanjee +22
A seminal paper published by Ledley and Lusted in 1959 introduced complex clinical diagnostic reasoning cases as the gold standard for the evaluation of expert medical computing sy…
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Scott L. Fleming, Alejandro Lozano, William J. Haberkorn +27
The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burd…
Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery
Debadutta Dash, Rahul Thapa, Juan M. Banda +15
Despite growing interest in using large language models (LLMs) in healthcare, current explorations do not assess the real-world utility and safety of LLMs in clinical settings. Our…