3 citations · 5 across the 16 of their papers we have counts for
22 papers
Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis
Rupali Rajesh, Sethuraman T, Savya Khosla +4
Audio-language models remain unreliable at extracting basic musical properties from audio, motivating pipelines that place a music information retrieval (MIR) tool between the audi…
Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs
Yuqun Wu, Yao Xiao, Chuhang Zou +2
Recent works augment Vision-Language Models with geometry features from pretrained 3D models, expecting that the geometric signal will boost spatial reasoning. However, we find tha…
SAM-V: Geometry-Aware Segment Anything for Multi-View Instance Segmentation
Jiangshan Gong, Yuqun Wu, Qiqian Fu +4
Consistent multi-view object segmentation is critical for 3D perception and robotics, yet remains challenging under severe viewpoint and occlusion changes. Existing methods typical…
ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Edits
Sethuraman T, Savya Khosla, Onkar Kishor Susladkar +6
Video-text models adapted from image-text architectures (e.g., CLIP) frequently exhibit temporal blindness, the inability to perceive fundamental cues like order, direction, and mo…
Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation
Prachi Garg, Steve Xing, Prahit Yaugand +2
State-of-the-art vision-language-action (VLA) models such as exhibit strong semantic understanding, instruction following and task behavior. However, when deployed on new…
ReToken: Improving Long-Context VLMs with Visual Retrieval Token
Yao Xiao, Reuben Tan, Zhen Zhu +3
Long visual contexts challenge vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once can exceed GPU memory limits. We p…
Decomposing Queries into Tool Calls for Long-Video Keyframe Retrieval
Michal Shlapentokh-Rothman, Prachi Garg, Yu-Xiong Wang +1
Keyframe selection is a direct way to provide verifiable visual evidence for long-video question answering (QA). Queries differ in what they require, and finding the right frames d…
T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability
Savya Khosla, Sethuraman T, Aryan Chadha +2
Despite recent progress, vision-language encoders struggle with two core limitations: (1) weak alignment between language and dense vision features, which hurts tasks like open-voc…
Why Fine-Tuning Encourages Hallucinations and How to Fix It
Guy Kaplan, Zorik Gekhman, Zhen Zhu +5
Large language models are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning…
Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
Sethuraman T, Savya Khosla, Aditi Tiwari +12
Video-Language Models (VidLMs) achieve strong benchmark scores, yet these scores often hide whether models use the video at all. We show that VidLM failures follow two pathways: so…
SceneDiff: A Benchmark and Method for Multiview Object Change Detection
Yuqun Wu, Chih-hao Lin, Henry Che +4
We investigate the problem of identifying objects that have been added, removed, or moved between a pair of captures (images or videos) of the same scene at different times. Accura…
How to Teach Large Multimodal Models New Skills
Zhen Zhu, Yiming Gong, Yao Xiao +2
How can we teach large multimodal models (LMMs) new skills without erasing prior abilities? We study sequential fine-tuning on five target skills while monitoring general ability o…
FRAME: Pre-Training Video Feature Representations via Anticipation and Memory
Sethuraman TV, Savya Khosla, Vignesh Srinivasakumar +5
Dense video prediction tasks, such as object tracking and semantic segmentation, require video encoders that generate temporally consistent, spatially dense features for every fram…
TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
Yao Xiao, Qiqian Fu, Heyi Tao +3
Image-text models excel at image-level tasks but struggle with detailed visual understanding. While these models provide strong visual-language alignment, segmentation models like…
PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding
Ansel Blume, Jeonghwan Kim, Hyeonjeong Ha +7
Real-world objects are composed of distinctive, object-specific parts. Identifying these parts is key to performing fine-grained, compositional reasoning-yet, large multimodal mode…
REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders
Savya Khosla, Sethuraman TV, Barnett Lee +2
We introduce the Region Encoder Network (REN), a fast and effective model for generating region-based image representations using point prompts. Recent methods combine class-agnost…
Visual Program Distillation with Template-Based Augmentation
Michal Shlapentokh-Rothman, Yu-Xiong Wang, Derek Hoiem
Adapting visual programming or prompting large language models (LLMs) to generate executable code for visual tasks like visual question answering (VQA) for specialized tasks or dom…
Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB
Jae Yong Lee, Yuqun Wu, Chuhang Zou +2
The goal of this paper is to encode a 3D scene into an extremely compact representation from 2D images and to enable its transmittance, decoding and rendering in real-time across v…
MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based Monocular Guidance
Yuqun Wu, Jae Yong Lee, Chuhang Zou +2
The latest regularized Neural Radiance Field (NeRF) approaches produce poor geometry and view extrapolation for large scale sparse view scenes, such as ETH3D. Density-based approac…
Region-Based Representations Revisited
Michal Shlapentokh-Rothman, Ansel Blume, Yao Xiao +7
We investigate whether region-based representations are effective for recognition. Regions were once a mainstay in recognition approaches, but pixel and patch-based features are no…
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Jiasen Lu, Christopher Clark, Sangho Lee +5
We present Unified-IO 2, the first autoregressive multimodal model that is capable of understanding and generating image, text, audio, and action. To unify different modalities, we…
WebWISE: Web Interface Control and Sequential Exploration with Large Language Models
Heyi Tao, Sethuraman T, Michal Shlapentokh-Rothman +1
The paper investigates using a Large Language Model (LLM) to automatically perform web software tasks using click, scroll, and text input operations. Previous approaches, such as r…