8 papers
Tracking by Predicting 3-D Gaussians Over Time
Tanish Baranwal, Himanshu Gaurav Singh, Jathushan Rajasegaran +1
We propose Video Gaussian Masked Autoencoders (Video-GMAE), a self-supervised approach for representation learning that encodes a sequence of images into a set of Gaussian splats m…
FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos
Yulu Gan, Ligeng Zhu, Dandan Shan +8
Motion understanding is fundamental to physical reasoning, enabling models to infer dynamics and predict future states. However, state-of-the-art models still struggle on recent mo…
Wolf: Dense Video Captioning with a World Summarization Framework
Boyi Li, Ligeng Zhu, Ran Tian +20
We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging comp…
Poly-Autoregressive Prediction for Modeling Interactions
Neerja Thakkar, Tara Sadjadpour, Jathushan Rajasegaran +2
We introduce a simple framework for predicting the behavior of an agent in multi-agent settings. In contrast to autoregressive (AR) tasks, such as language processing, our focus is…
Interactive Task Planning with Language Models
Boyi Li, Philipp Wu, Pieter Abbeel +1
An interactive robot framework accomplishes long-horizon task planning and can easily generalize to new goals and distinct tasks, even during execution. However, most traditional m…
An Empirical Study of Autoregressive Pre-training from Videos
Jathushan Rajasegaran, Ilija Radosavovic, Rahul Ravishankar +3
We empirically study autoregressive pre-training from videos. To perform our study, we construct a series of autoregressive video models, called Toto. We treat videos as sequences…