15 papers · 1 filter
Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents
Xueqiao Sun, Xiaohan Wang, Ludwig Schmidt +2
Computer-use agents, which leverage multimodal large language models (MLLMs) to operate computers and complete tasks, have attracted significant attention for their utility and ver…
Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
He Li, Yuhui Zhang, Xiaohan Wang +2
The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual qu…
RadDiff: Describing Differences in Radiology Image Sets with Natural Language
Xiaoxian Shen, Yuhui Zhang, Sahithi Ankireddy +5
Understanding how two radiology image sets differ is critical for generating clinical insights and for interpreting medical AI systems. We introduce RadDiff, a multimodal agentic s…
Transductive Visual Programming: Evolving Tool Libraries from Experience for Spatial Reasoning
Shengguang Wu, Xiaohan Wang, Yuhui Zhang +2
Spatial reasoning in 3D scenes requires precise geometric calculations that challenge vision-language models. Visual programming addresses this by decomposing problems into steps c…
Temporal Preference Optimization for Long-Form Video Understanding
Rui Li, Xiaohan Wang, Yuhui Zhang +3
Despite significant advancements in video large multimodal models (video-LMMs), achieving effective temporal grounding in long-form videos remains a challenge for existing models.…
Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
Mark Endo, Xiaohan Wang, Serena Yeung-Levy
Recent works on accelerating Vision-Language Models achieve strong performance across a variety of vision-language tasks despite highly compressing visual information. In this work…