77 citations · 97 across the 23 of their papers we have counts for
26 papers
Can Language Models Identify Side Effects of Breast Cancer Radiation Treatments?
Natalie Seah, Danielle S. Bitterman, Daphna Spiegel +1
Accurately communicating the side effects of cancer treatments to cancer survivors is critical, particularly in settings such as informed consent, where clinicians must clearly and…
Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments
Bingyang Ye, Shan Chen, Jingxuan Tu +4
Large language models are increasingly being used to assess and forecast research ideas, yet we lack scalable ways to evaluate the quality of models' judgments about these scientif…
Large language models require a new form of oversight: capability-based monitoring
Katherine C. Kellogg, Bingyang Ye, Yifan Hu +3
The rapid adoption of large language models (LLMs) in healthcare has been accompanied by scrutiny of their oversight. Existing monitoring approaches, inherited from traditional mac…
A Field Guide to Deploying AI Agents in Clinical Practice
Jack Gallifant, Katherine C. Kellogg, Matt Butler +19
Large language models (LLMs) integrated into agent-driven workflows hold immense promise for healthcare, yet a significant gap exists between their potential and practical implemen…
MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
Shan Chen, Pedro Moreira, Yuxin Xiao +6
Large language models (LLMs) are increasingly envisioned as decision-support tools in clinical practice, yet safe clinical reasoning demands integrating heterogeneous knowledge bas…
When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy
Jirui Qi, Shan Chen, Zidi Xiong +3
Recent Large Reasoning Models (LRMs) with thinking traces have shown strong performance on English reasoning tasks. However, their ability to think in other languages is less studi…
TheBlueScrubs-v1, a comprehensive curated medical dataset derived from the internet
Luis Felipe, Carlos Garcia, Issam El Naqa +6
The need for robust and diverse data sets to train clinical large language models (cLLMs) is critical given that currently available public repositories often prove too limited in…
Sparse Autoencoder Features for Classifications and Transferability
Jack Gallifant, Shan Chen, Kuleen Sasse +3
Sparse Autoencoders (SAEs) provide potentials for uncovering structured, human-interpretable representations in Large Language Models (LLMs), making them a crucial tool for transpa…
Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs
David Restrepo, Chenwei Wu, Zhengxu Tang +14
Current ophthalmology clinical workflows are plagued by over-referrals, long waits, and complex and heterogeneous medical records. Large language models (LLMs) present a promising…
The use of large language models to enhance cancer clinical trial educational materials
Mingye Gao, Aman Varshney, Shan Chen +15
Cancer clinical trials often face challenges in recruitment and engagement due to a lack of participant-facing informational and educational resources. This study investigated the…
Position Paper On Diagnostic Uncertainty Estimation from Large Language Models: Next-Word Probability Is Not Pre-test Probability
Yanjun Gao, Skatje Myers, Shan Chen +7
Large language models (LLMs) are being explored for diagnostic decision support, yet their ability to estimate pre-test probabilities, vital for clinical decision-making, remains l…
ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?
Canyu Chen, Jian Yu, Shan Chen +8
Large Language Models (LLMs) hold great promise to revolutionize current clinical systems for their superior capacities on medical text processing tasks and medical licensing exams…
WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
João Matos, Shan Chen, Siena Placino +13
Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fai…
debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
Kuleen Sasse, Shan Chen, Jackson Pond +2
As Vision Language Models (VLMs) gain widespread use, their fairness remains under-explored. In this paper, we analyze demographic biases across five models and six datasets. We fi…
Wait, but Tylenol is Acetaminophen... Investigating and Improving Language Models' Ability to Resist Requests for Misinformation
Shan Chen, Mingye Gao, Kuleen Sasse +6
Background: Large language models (LLMs) are trained to follow directions, but this introduces a vulnerability to blindly comply with user requests even if they generate wrong info…
When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?
Yanjun Gao, Skatje Myers, Shan Chen +5
The introduction of Large Language Models (LLMs) has advanced data representation and analysis, bringing significant progress in their use for medical questions and answering. Desp…
Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education
Huizi Yu, Jiayan Zhou, Lingyao Li +22
Background: Simulated patient systems are important in medical education and research, providing safe, integrative training environments and supporting clinical decision making. Ad…
Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
Shan Chen, Jack Gallifant, Mingye Gao +12
Large language models (LLMs) are increasingly essential in processing natural languages, yet their application is frequently compromised by biases and inaccuracies originating in t…
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
Jack Gallifant, Shan Chen, Pedro Moreira +7
Medical knowledge is context-dependent and requires consistent reasoning across various natural language expressions of semantically equivalent phrases. This is particularly crucia…
Retrieval-Augmented Generation for Generative Artificial Intelligence in Medicine
Rui Yang, Yilin Ning, Emilia Keppo +6
Generative artificial intelligence (AI) has brought revolutionary innovations in various fields, including medicine. However, it also exhibits limitations. In response, retrieval-a…
Analyzing Diversity in Healthcare LLM Research: A Scientometric Perspective
David Restrepo, Chenwei Wu, Constanza Vásquez-Venegas +5
The deployment of large language models (LLMs) in healthcare has demonstrated substantial potential for enhancing clinical decision-making, administrative efficiency, and patient o…
Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources
Lasse Hyldig Hansen, Nikolaj Andersen, Jack Gallifant +7
Background Advancements in Large Language Models (LLMs) hold transformative potential in healthcare, however, recent work has raised concern about the tendency of these models to p…
Improving Clinical NLP Performance through Language Model-Generated Synthetic Clinical Data
Shan Chen, Jack Gallifant, Marco Guevara +5
Generative models have been showing potential for producing data in mass. This study explores the enhancement of clinical natural language processing performance by utilizing synth…
Measuring Pointwise -Usable Information In-Context-ly
Sheng Lu, Shan Chen, Yingya Li +3
In-context learning (ICL) is a new learning paradigm that has gained popularity along with the development of large language models. In this work, we adapt a recently proposed hard…