6 papers
Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang +3
Vision-Language Models (VLMs) often struggle with robust 3D spatial reasoning. Prevailing methods that rely on fine-tuning with 3D visual question-answering (VQA) datasets may over…
Computer Use at the Edge of the Statistical Precipice
Pierluca D'Oro, Sneha Silwal, William Wong +6
Evaluating Computer Use Agents (CUAs) on interactive environments is fraught with methodological pitfalls that the field has yet to systematically address. We show that a 1MB repla…
DigiData: Training and Evaluating General-Purpose Mobile Control Agents
Yuxuan Sun, Manchen Wang, Shengyi Qian +18
AI agents capable of controlling user interfaces have the potential to transform human interaction with digital devices. To accelerate this transformation, two fundamental building…
Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents
Vijay Veerabadran, Fanyi Xiao, Nitin Kamra +10
There has been a surge of interest in assistive wearable agents: agents embodied in wearable form factors (e.g., smart glasses) who take assistive actions toward a user's goal/quer…
Enhancing Visual Planning with Auxiliary Tasks and Multi-token Prediction
Ce Zhang, Yale Song, Ruta Desai +4
Visual Planning for Assistance (VPA) aims to predict a sequence of user actions required to achieve a specified goal based on a video showing the user's progress. Although recent a…
Embodied AI Agents: Modeling the World
Pascale Fung, Yoram Bachrach, Asli Celikyilmaz +18
This paper describes our research on AI agents embodied in visual, virtual or physical forms, enabling them to interact with both users and their environments. These agents, which…