4 papers · 1 filter
Sequential Diagnosis with Language Models
Harsha Nori, Mayank Daswani, Christopher Kelly +12
Artificial intelligence holds great promise for expanding access to expert medical knowledge and reasoning. However, most evaluations of language models rely on static vignettes an…
JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
Saibo Geng, Hudson Cooper, MichaÅ Moskal +6
Reliably generating structured outputs has become a critical capability for modern language model (LM) applications. Constrained decoding has emerged as the dominant technology acr…
From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond
Harsha Nori, Naoto Usuyama, Nicholas King +4
Run-time steering strategies like Medprompt are valuable for guiding large language models (LLMs) to top performance on challenging tasks. Medprompt demonstrates that a general LLM…
MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering
Robert Osazuwa Ness, Katie Matton, Hayden Helm +4
Large language models (LLM) have achieved impressive performance on medical question-answering benchmarks. However, high benchmark accuracy does not imply that the performance gene…