40 citations · 59 across the 66 of their papers we have counts for
19 papers · 1 filter
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Orr Zohar, Xiaohan Wang, Yann Dubois +9
Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly unders…
Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
Mark Endo, Xiaohan Wang, Serena Yeung-Levy
Recent works on accelerating Vision-Language Models achieve strong performance across a variety of vision-language tasks despite highly compressing visual information. In this work…
DeforHMR: Vision Transformer with Deformable Cross-Attention for 3D Human Mesh Recovery
Jaewoo Heo, George Hu, Zeyu Wang +1
Human Mesh Recovery (HMR) is an important yet challenging problem with applications across various domains including motion capture, augmented reality, and biomechanics. Accurately…
Motion Diffusion-Guided 3D Global HMR from a Dynamic Camera
Jaewoo Heo, Kuan-Chieh Wang, Karen Liu +1
Motion capture technologies have transformed numerous fields, from the film and gaming industries to sports science and healthcare, by providing a tool to capture and analyze human…
How to Build the Virtual Cell with Artificial Intelligence: Priorities and Opportunities
Charlotte Bunne, Yusuf Roohani, Yanay Rosen +39
The cell is arguably the most fundamental unit of life and is central to understanding biology. Accurate modeling of cells is important for this understanding as well as for determ…
Zero-shot Action Localization via the Confidence of Large Vision-Language Models
Josiah Aklilu, Xiaohan Wang, Serena Yeung-Levy
Precise action localization in untrimmed video is vital for fields such as professional sports and minimally invasive surgery, where the delineation of particular motions in record…
Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Models
Laura Bravo-Sánchez, Jaewoo Heo, Zhenzhen Weng +2
Social dynamics in close human interactions pose significant challenges for Human Mesh Estimation (HME), particularly due to the complexity of physical contacts and the scarcity of…
Continuous Perception Matters: Diagnosing Temporal Integration Failures in Multimodal Models
Zeyu Wang, Zhenzhen Weng, Serena Yeung-Levy
Continuous perception, the ability to integrate visual observations over time in a continuous stream fashion, is essential for robust real-world understanding, yet remains largely…
Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision
Orr Zohar, Xiaohan Wang, Yonatan Bitton +2
The performance of Large Vision Language Models (LVLMs) is dependent on the size and quality of their training datasets. Existing video instruction tuning datasets lack diversity a…
μ-Bench: A Vision-Language Benchmark for Microscopy Understanding
Alejandro Lozano, Jeffrey Nirschl, James Burgess +4
Recent advances in microscopy have enabled the rapid generation of terabytes of image data in cell biology and biomedical research. Vision-language models (VLMs) offer a promising…
Why are Visually-Grounded Language Models Bad at Image Classification?
Yuhui Zhang, Alyssa Unell, Xiaohan Wang +4
Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visually-grounded lang…
VideoAgent: Long-form Video Understanding with Large Language Model as Agent
Xiaohan Wang, Yuhui Zhang, Orr Zohar +1
Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the hu…
Depth-guided NeRF Training via Earth Mover's Distance
Anita Rau, Josiah Aklilu, F. Christopher Holsinger +1
Neural Radiance Fields (NeRFs) are trained to minimize the rendering loss of predicted viewpoints. However, the photometric loss often does not provide enough information to disamb…
Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models
Elaine Sui, Xiaohan Wang, Serena Yeung-Levy
Advancements in vision-language models (VLMs) have propelled the field of computer vision, particularly in the zero-shot learning setting. Despite their promise, the effectiveness…
Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation
Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu +24
The scaling laws and extraordinary performance of large foundation models motivate the development and utilization of such models in biomedicine. However, despite early promising r…
Multi-Human Mesh Recovery with Transformers
Zeyu Wang, Zhenzhen Weng, Serena Yeung-Levy
Conventional approaches to human mesh recovery predominantly employ a region-based strategy. This involves initially cropping out a human-centered region as a preprocessing step, w…
Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data
Yuhui Zhang, Elaine Sui, Serena Yeung-Levy
Building cross-modal applications is challenging due to limited paired multi-modal data. Recent works have shown that leveraging a pre-trained multi-modal contrastive representatio…
Revisiting Active Learning in the Era of Vision Foundation Models
Sanket Rajan Gupte, Josiah Aklilu, Jeffrey J. Nirschl +1
Foundation vision or vision-language models are trained on large unlabeled or noisy data and learn robust representations that can achieve impressive zero- or few-shot performance…
Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM
Zhenzhen Weng, Jingyuan Liu, Hao Tan +4
Reconstructing 3D humans from a single image has been extensively investigated. However, existing approaches often fall short on capturing fine geometry and appearance details, hal…