40 citations · 59 across the 66 of their papers we have counts for
25 papers · 1 filter
Transductive Visual Programming: Evolving Tool Libraries from Experience for Spatial Reasoning
Shengguang Wu, Xiaohan Wang, Yuhui Zhang +2
Spatial reasoning in 3D scenes requires precise geometric calculations that challenge vision-language models. Visual programming addresses this by decomposing problems into steps c…
CryoHype: Reconstructing a thousand cryo-EM structures with transformer-based hypernetworks
Jeffrey Gu, Minkyu Jeon, Ambri Ma +2
Cryo-electron microscopy (cryo-EM) is an indispensable technique for determining the 3D structures of dynamic biomolecular complexes. While typically applied to image a single mole…
From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
Kun Yuan, Min Woo Sun, Zhen Chen +7
There is a growing interest in developing strong biomedical vision-language models. A popular approach to achieve robust representations is to use web-scale scientific data. Howeve…
Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models
Mark Endo, Serena Yeung-Levy
Scaling up multimodal models has enabled remarkable advances in visual understanding and reasoning, but practical demands call for smaller, efficient systems. In this work, we cond…
The Impact of Image Resolution on Biomedical Multimodal Large Language Models
Liangyu Chen, James Burgess, Jeffrey J Nirschl +2
Imaging technologies are fundamental to biomedical research and modern medicine, requiring analysis of high-resolution images across various modalities. While multimodal large lang…
Data or Language Supervision: What Makes CLIP Better than DINO?
Yiming Liu, Yuhui Zhang, Dhruba Ghosh +2
CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs), but it remains unclear whether this advantage stems from CLIP's language sup…
SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models
Andong Deng, Taojiannan Yang, Shoubin Yu +5
Large Multimodal Models (LMMs) have achieved remarkable progress across various capabilities; however, complex video reasoning in the scientific domain remains a significant and ch…
Three Forms of Stochastic Injection for Improved Distribution-to-Distribution Generative Modeling
Shiye Su, Yuhui Zhang, Linqi Zhou +2
Modeling transformations between arbitrary data distributions is a fundamental scientific challenge, arising in applications like drug discovery and evolutionary simulation. While…
No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models
Min Woo Sun, Alejandro Lozano, Javier Gamazo Tejero +8
Embedding vision-language models (VLMs) are typically pretrained with short text windows (<77 tokens), which forces the truncation of long-format captions. Yet, the distribution of…
TTRV: Test-Time Reinforcement Learning for Vision Language Models
Akshit Singh, Shyam Marjit, Wei Lin +7
Existing methods for extracting reward signals in Reinforcement Learning typically rely on labeled data and dedicated training splits, a setup that contrasts with how humans learn…
AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond
Shangding Gu, Xiaohan Wang, Donghao Ying +9
Rapid advances in multimodal models demand benchmarks that rigorously evaluate understanding and reasoning in safety-critical, dynamic real-world settings. We present AccidentBench…
VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
Paul Gavrikov, Wei Lin, M. Jehanzeb Mirza +6
Is basic visual understanding really solved in state-of-the-art VLMs? We present VisualOverload, a slightly different visual question answering (VQA) benchmark comprising 2,720 que…
Closing the Modality Gap for Mixed Modality Search
Binxu Li, Yuhui Zhang, Xiaohan Wang +3
Mixed modality search -- retrieving information across a heterogeneous corpus composed of images, texts, and multimodal documents -- is an important yet underexplored real-world ap…
NegVQA: Can Vision Language Models Understand Negation?
Yuhui Zhang, Yuchang Su, Yiming Liu +1
Negation is a fundamental linguistic phenomenon that can entirely reverse the meaning of a sentence. As vision language models (VLMs) continue to advance and are deployed in high-s…
Can Large Language Models Match the Conclusions of Systematic Reviews?
Christopher Polzak, Alejandro Lozano, Min Woo Sun +4
Systematic reviews (SR), in which experts summarize and analyze evidence across individual studies to provide insights on a specialized topic, are a cornerstone for evidence-based…
Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence
Anita Rau, Mark Endo, Josiah Aklilu +6
Large Vision-Language Models offer a new paradigm for AI-driven image understanding, enabling models to perform tasks without task-specific training. This flexibility holds particu…
A Large-Scale Vision-Language Dataset Derived from Open Scientific Literature to Advance Biomedical Generalist AI
Alejandro Lozano, Min Woo Sun, James Burgess +16
Despite the excitement behind biomedical artificial intelligence (AI), access to high-quality, diverse, and large-scale data - the foundation for modern AI systems - is still a bot…
MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
James Burgess, Jeffrey J Nirschl, Laura Bravo-Sánchez +20
Scientific research demands sophisticated reasoning over multimodal data, a challenge especially prevalent in biology. Despite recent advances in multimodal large language models (…
Video Action Differencing
James Burgess, Xiaohan Wang, Yuhui Zhang +5
How do two individuals differ when performing the same action? In this work, we introduce Video Action Differencing (VidDiff), the novel task of identifying subtle differences betw…
SurgiSAM2: Fine-tuning a foundational model for surgical video anatomy segmentation and detection
Devanish N. Kamtam, Joseph B. Shrager, Satya Deepya Malla +5
Background: We evaluate SAM 2 for surgical scene understanding by examining its semantic segmentation capabilities for organs/tissues both in zero-shot scenarios and after fine-tun…
Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models
Jeffrey Gu, Serena Yeung-Levy
Large pre-trained models, or foundation models, have shown impressive performance when adapted to a variety of downstream tasks, often out-performing specialized models. Hypernetwo…
CellFlux: Simulating Cellular Morphology Changes via Flow Matching
Yuhui Zhang, Yuchang Su, Chenyu Wang +8
Building a virtual cell capable of accurately simulating cellular behaviors in silico has long been a dream in computational biology. We introduce CellFlux, an image-generative mod…
Temporal Preference Optimization for Long-Form Video Understanding
Rui Li, Xiaohan Wang, Yuhui Zhang +3
Despite significant advancements in video large multimodal models (video-LMMs), achieving effective temporal grounding in long-form videos remains a challenge for existing models.…
BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature
Alejandro Lozano, Min Woo Sun, James Burgess +13
The development of vision-language models (VLMs) is driven by large-scale and diverse multimodal datasets. However, progress toward generalist biomedical VLMs is limited by the lac…