Publications (31)
Language models scale reliably with over-training and on downstream tasks
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar +22
Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps…
Zero-Shot Open-Vocabulary Tracking with Large Pre-Trained Models
Wen-Hsuan Chu, Adam W. Harley, Pavel Tokmakov +3
Object tracking is central to robot perception and scene understanding. Tracking-by-detection has long been a dominant paradigm for object tracking of specific object categories. R…
Understanding Video Transformers via Universal Concept Discovery
Matthew Kowal, Achal Dave, Rares Ambrus +3
This paper studies the problem of concept-based interpretability of transformer representations for videos. Concretely, we seek to explain the decision-making process of video tran…
Do Image Classifiers Generalize Across Time?
Vaishaal Shankar, Achal Dave, Rebecca Roelofs +3
We study the robustness of image classifiers to temporal perturbations derived from videos. As part of this study, we construct two datasets, ImageNet-Vid-Robust and YTBB-Robust ,…
Learning to Track Any Object
Achal Dave, Pavel Tokmakov, Cordelia Schmid +1
Object tracking can be formulated as "finding the right object in a video". We observe that recent approaches for class-agnostic tracking tend to focus on the "finding" part, but l…
Understanding Complexity in VideoQA via Visual Program Generation
Cristobal Eyzaguirre, Igor Vasiljevic, Achal Dave +5
We propose a data-driven approach to analyzing query complexity in Video Question Answering (VideoQA). Previous efforts in benchmark design have relied on human expertise to design…
Long-Tailed 3D Detection via Multi-Modal Fusion
Yechi Ma, Neehar Peri, Achal Dave +3
Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors. While class labels naturally follow a long-tailed distribution in the real world…
Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details
Achal Dave, Piotr Dollár, Deva Ramanan +2
By design, average precision (AP) for object detection aims to treat all classes independently: AP is computed independently per category and averaged. On one hand, this is desirab…
pix2gestalt: Amodal Segmentation by Synthesizing Wholes
Ege Ozguroglu, Ruoshi Liu, DÃdac SurÃs +4
We introduce pix2gestalt, a framework for zero-shot amodal segmentation, which learns to estimate the shape and appearance of whole objects that are only partially visible behind o…
Opening up Open-World Tracking
Yang Liu, Idil Esen Zulfikar, Jonathon Luiten +5
Tracking and detecting any object, including ones never-seen-before during model training, is a crucial but elusive capability of autonomous systems. An autonomous agent that is bl…
OpenThoughts: Data Recipes for Reasoning Models
Etash Guha, Ryan Marten, Sedrick Keh +47
Reasoning models have made rapid progress on many benchmarks involving math, code, and science. Yet, there are still many open questions about the best training recipes for reasoni…
DataComp-LM: In search of the next generation of training sets for language models
Jeffrey Li, Alex Fang, Georgios Smyrnis +56
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardize…
Towards Segmenting Anything That Moves
Achal Dave, Pavel Tokmakov, Deva Ramanan
Detecting and segmenting individual objects, regardless of their category, is crucial for many applications such as action detection or robotic interaction. While this problem has…
Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis
Basile Van Hoorick, Rundi Wu, Ege Ozguroglu +6
Accurate reconstruction of complex dynamic scenes from just a single viewpoint continues to be a challenging task in computer vision. Current dynamic novel view synthesis methods t…
Espresso: High Compression For Rich Extraction From Videos for Your Vision-Language Model
Keunwoo Peter Yu, Achal Dave, Rares Ambrus +1
Recent advances in vision-language models (VLMs) have shown great promise in connecting images and text, but extending these models to long videos remains challenging due to the ra…
Differentiable Raycasting for Self-supervised Occupancy Forecasting
Tarasha Khurana, Peiyun Hu, Achal Dave +3
Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene no…
AllTracker: Efficient Dense Point Tracking at High Resolution
Adam W. Harley, Yang You, Xinglong Sun +11
We introduce AllTracker: a model that estimates long-range point tracks by way of estimating the flow field between a query frame and every other frame of a video. Unlike existing…
Linearizing Large Language Models
Jean Mercat, Igor Vasiljevic, Sedrick Keh +4
Linear transformers have emerged as a subquadratic-time alternative to softmax attention and have garnered significant interest due to their fixed-size recurrent state that lowers…
Predictive-Corrective Networks for Action Detection
Achal Dave, Olga Russakovsky, Deva Ramanan
While deep feature learning has revolutionized techniques for static-image understanding, the same does not quite hold for video processing. Architectures and optimization techniqu…
TAO-Amodal: A Benchmark for Tracking Any Object Amodally
Cheng-Yen Hsieh, Kaihua Chen, Achal Dave +2
Amodal perception, the ability to comprehend complete object structures from partial visibility, is a fundamental skill, even for infants. Its significance extends to applications…
Towards Long-Tailed 3D Detection
Neehar Peri, Achal Dave, Deva Ramanan +1
Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors, particularly on large-scale lidar data. Surprisingly, although semantic class la…
Shape of You: Precise 3D shape estimations for diverse body types
Rohan Sarkar, Achal Dave, Gerard Medioni +1
This paper presents Shape of You (SoY), an approach to improve the accuracy of 3D body shape estimation for vision-based clothing recommendation systems. While existing methods hav…
Detecting Invisible People
Tarasha Khurana, Achal Dave, Deva Ramanan
Monocular object detection and tracking have improved drastically in recent years, but rely on a key assumption: that objects are visible to the camera. Many offline tracking appro…
Should VLMs be Pre-trained with Image Data?
Sedrick Keh, Jean Mercat, Samir Yitzhak Gadre +8
Pre-trained LLMs that are further trained with image data perform well on vision-language tasks. While adding images during a second training phase effectively unlocks this capabil…
HandsOff: Labeled Dataset Generation With No Additional Human Annotations
Austin Xu, Mariya I. Vasileva, Achal Dave +1
Recent work leverages the expressive power of generative adversarial networks (GANs) to generate labeled synthetic datasets. These dataset generation methods often require new anno…
BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video
Ali Athar, Jonathon Luiten, Paul Voigtlaender +4
Multiple existing benchmarks involve tracking and segmenting objects in video e.g., Video Object Segmentation (VOS) and Multi-Object Tracking and Segmentation (MOTS), but there is…
GRIN: Zero-Shot Metric Depth with Pixel-Level Diffusion
Vitor Guizilini, Pavel Tokmakov, Achal Dave +1
3D reconstruction from a single image is a long-standing problem in computer vision. Learning-based methods address its inherent scale ambiguity by leveraging increasingly large la…
TAO: A Large-Scale Benchmark for Tracking Any Object
Achal Dave, Tarasha Khurana, Pavel Tokmakov +2
For many years, multi-object tracking benchmarks have focused on a handful of categories. Motivated primarily by surveillance and self-driving applications, these datasets provide…
Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
Junbang Liang, Ruoshi Liu, Ege Ozguroglu +5
A key challenge in manipulation is learning a policy that can robustly generalize to diverse visual environments. A promising mechanism for learning robust policies is to leverage…
Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)
Alex Fang, Gabriel Ilharco, Mitchell Wortsman +4
Contrastively trained language-image models such as CLIP, ALIGN, and BASIC have demonstrated unprecedented robustness to multiple challenging natural distribution shifts. Since the…
Measuring Robustness to Natural Distribution Shifts in Image Classification
Rohan Taori, Achal Dave, Vaishaal Shankar +3
We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. Most research on robustness focuses on synthetic image perturbat…