1.2k citations · 2.1k across the 41 of their papers we have counts for
46 papers
Scaling medical imaging report generation with multimodal reinforcement learning
Qianchu Liu, Sheng Zhang, Guanghui Qin +10
Frontier models have demonstrated remarkable capabilities in understanding and reasoning with natural-language text, but they still exhibit major competency gaps in multimodal unde…
OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning
Timothy Ossowski, Sheng Zhang, Qianchu Liu +5
High-quality and carefully curated data is a cornerstone of training medical large language models, as it directly impacts both generalization and robustness to unseen clinical tas…
Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaboration
James Y. Huang, Sheng Zhang, Qianchu Liu +5
Large Language Models (LLMs) have demonstrated remarkable capabilities in challenging, knowledge-intensive reasoning tasks. However, extending LLMs to perceive and reason over a ne…
ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation
Qin Liu, Jacob Dineen, Yuxi Huang +4
Benchmarks are central to measuring the capabilities of large language models and guiding model development, yet widespread data leakage from pretraining corpora undermines their v…
X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains
Qianchu Liu, Sheng Zhang, Guanghui Qin +9
Recent proprietary models (e.g., o3) have begun to demonstrate strong multimodal reasoning capabilities. Yet, most existing open-source research concentrates on training text-only…
Exploring Scaling Laws for EHR Foundation Models
Sheng Zhang, Qin Liu, Naoto Usuyama +3
The emergence of scaling laws has profoundly shaped the development of large language models (LLMs), enabling predictable performance gains through systematic increases in model si…
MetaScale: Test-Time Scaling with Evolving Meta-Thoughts
Qin Liu, Wenxuan Zhou, Nan Xu +5
One critical challenge for large language models (LLMs) for making complex reasoning is their reliance on matching reasoning patterns from training data, instead of proactively sel…
Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection
Bangzheng Li, Fei Wang, Wenxuan Zhou +5
Vision-Language Models (VLMs) leverage aligned visual encoders to transform images into visual tokens, allowing them to be processed similarly to text by the backbone large languag…
Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learning
Sheng Zhang, Qianchu Liu, Guanghui Qin +2
Reinforcement learning from verifiable rewards (RLVR) has recently gained attention for its ability to elicit self-evolved reasoning capabilitie from base language models without e…
Universal Abstraction: Harnessing Frontier Models to Structure Real-World Data at Scale
Cliff Wong, Sam Preston, Qianchu Liu +22
A significant fraction of real-world patient information resides in unstructured clinical text. Medical abstraction extracts and normalizes key structured attributes from free-text…
From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond
Harsha Nori, Naoto Usuyama, Nicholas King +4
Run-time steering strategies like Medprompt are valuable for guiding large language models (LLMs) to top performance on challenging tasks. Medprompt demonstrates that a general LLM…
MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging
Noel C. F. Codella, Ying Jin, Shrey Jain +28
In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a dive…
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Fei Wang, Xingyu Fu, James Y. Huang +18
We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tas…
From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
Nan Xu, Fei Wang, Sheng Zhang +2
Motivated by in-context learning (ICL) capabilities of Large Language Models (LLMs), multimodal LLMs with additional visual modality are also exhibited with similar ICL abilities w…
mDPO: Conditional Preference Optimization for Multimodal Large Language Models
Fei Wang, Wenxuan Zhou, James Y. Huang +4
Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios…
MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering
Robert Osazuwa Ness, Katie Matton, Hayden Helm +4
Large language models (LLM) have achieved impressive performance on medical question-answering benchmarks. However, high benchmark accuracy does not imply that the performance gene…
Offset Unlearning for Large Language Models
James Y. Huang, Wenxuan Zhou, Fei Wang +4
Despite the strong capabilities of Large Language Models (LLMs) to acquire knowledge from their training corpora, the memorization of sensitive information in the corpora such as c…
Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation
Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu +24
The scaling laws and extraordinary performance of large foundation models motivate the development and utilization of such models in biomedicine. However, despite early promising r…
Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries
Zelalem Gero, Chandan Singh, Yiqing Xie +6
Summarizing clinical text is crucial in health decision-support and clinical research. Large language models (LLMs) have shown the potential to generate accurate clinical text summ…
T-Rex: Text-assisted Retrosynthesis Prediction
Yifeng Liu, Hanwen Xu, Tangqi Fang +5
As a fundamental task in computational chemistry, retrosynthesis prediction aims to identify a set of reactants to synthesize a target molecule. Existing template-free approaches o…
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Harsha Nori, Yin Tat Lee, Sheng Zhang +15
Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot matc…
DocLens: Multi-aspect Fine-grained Evaluation for Medical Text Generation
Yiqing Xie, Sheng Zhang, Hao Cheng +6
Medical text generation aims to assist with administrative work and highlight salient information to support decision-making. To reflect the specific requirements of medical text,…
BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys
Yu Gu, Jianwei Yang, Naoto Usuyama +5
Rapid progress has been made in instruction-learning for image editing with natural-language instruction, as exemplified by InstructPix2Pix. In biomedicine, such methods can be app…
Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology
Cliff Wong, Sheng Zhang, Yu Gu +8
Clinical trial matching is a key process in health delivery and discovery. In practice, it is plagued by overwhelming unstructured data and unscalable manual processing. In this pa…