28 papers
Self-Supervised Learning of Structured Dynamics from Videos
Lukas Knobel, Andrew Zisserman, Yuki M. Asano
Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This deco…
Video Generation Models are General-Purpose Vision Learners
Letian Wang, Chuhan Zhang, Rishabh Kabra +9
Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models. What, then, is the equivalent catalyst needed to achieve a genera…
Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing
Zifan Jiang, Youngjoon Jang, Liliane Momeni +3
The goal of this work is to develop a universal approach for aligning subtitles (i.e., spoken language text with corresponding timestamps) to continuous sign language videos. Prior…
WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata
Prasanna Sridhar, Horace Lee, David M. S. Pinto +2
In this paper, we present WISE, an open-source audiovisual search engine which integrates a range of multimodal retrieval capabilities into a single, practical tool accessible to u…
CountGD++: Generalized Prompting for Open-World Counting
Niki Amini-Naieni, Andrew Zisserman
The flexibility and accuracy of methods for automatically counting objects in images and videos are limited by the way the object can be specified. While existing methods allow use…
GMOS: Grounding Moving Object Segmentation in 3D Space and Time
Junyu Xie, Tengda Han, Weidi Xie +1
Moving Object Segmentation (MOS) aims to discover, segment, and track objects that move independently of the camera. Current MOS methods, however, exhibit two fundamental limitatio…