4 papers
DigiData: Training and Evaluating General-Purpose Mobile Control Agents
Yuxuan Sun, Manchen Wang, Shengyi Qian +18
AI agents capable of controlling user interfaces have the potential to transform human interaction with digital devices. To accelerate this transformation, two fundamental building…
Continual Dialogue State Tracking via Example-Guided Question Answering
Hyundong Cho, Andrea Madotto, Zhaojiang Lin +5
Dialogue systems are frequently updated to accommodate new services, but naively updating them by continually training with data for new services in diminishing performance on prev…
Synthetic Captions for Open-Vocabulary Zero-Shot Segmentation
Tim Lebailly, Vijay Veerabadran, Satwik Kottur +2
Generative vision-language models (VLMs) exhibit strong high-level image understanding but lack spatially dense alignment between vision and language modalities, as our findings in…
Enhancing Visual Planning with Auxiliary Tasks and Multi-token Prediction
Ce Zhang, Yale Song, Ruta Desai +4
Visual Planning for Assistance (VPA) aims to predict a sequence of user actions required to achieve a specified goal based on a video showing the user's progress. Although recent a…