7 citations · 24 across the 14 of their papers we have counts for
16 papers
SUAVE: Unified Video-Action Models via Masked Diffusion
Rhythm Syed, Jean Mercat, Sedrick Keh +5
Vision-language-action models (VLAs) inherit strong semantic grounding from pretrained vision-language backbones but are typically optimized for predicting actions rather than futu…
Cross-Embodiment Transfer via Behavior-Aligned Representations
Ajay Sridhar, Jensen Gao, Jonathan Yang +3
Recent progress in large-scale imitation learning for robot manipulation has been driven by leveraging datasets across a wide range of robot embodiments. However, achieving signifi…
VLA Foundry: A Unified Framework for Training Vision-Language-Action Models
Jean Mercat, Sedrick Keh, Kushal Arora +5
We present VLA Foundry, an open-source framework that unifies LLM, VLM, and VLA training in a single codebase. Most open-source VLA efforts specialize on the action training stage,…
A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation
Fanqi Lin, Kushal Arora, Jean Mercat +9
Large behavior models have shown strong dexterous manipulation capabilities by extending imitation learning to large-scale training on multi-task robot data, yet their generalizati…
A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation
TRI LBM Team, Jose Barreiros, Andrew Beaulieu +79
Robot manipulation has seen tremendous progress in recent years, with imitation learning policies enabling successful performance of dexterous and hard-to-model tasks. Concurrently…
OpenThoughts: Data Recipes for Reasoning Models
Etash Guha, Ryan Marten, Sedrick Keh +47
Reasoning models have made rapid progress on many benchmarks involving math, code, and science. Yet, there are still many open questions about the best training recipes for reasoni…
Should VLMs be Pre-trained with Image Data?
Sedrick Keh, Jean Mercat, Samir Yitzhak Gadre +8
Pre-trained LLMs that are further trained with image data perform well on vision-language tasks. While adding images during a second training phase effectively unlocks this capabil…
Espresso: High Compression For Rich Extraction From Videos for Your Vision-Language Model
Keunwoo Peter Yu, Achal Dave, Rares Ambrus +1
Recent advances in vision-language models (VLMs) have shown great promise in connecting images and text, but extending these models to long videos remains challenging due to the ra…
DataComp-LM: In search of the next generation of training sets for language models
Jeffrey Li, Alex Fang, Georgios Smyrnis +56
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardize…
Linearizing Large Language Models
Jean Mercat, Igor Vasiljevic, Sedrick Keh +4
Linear transformers have emerged as a subquadratic-time alternative to softmax attention and have garnered significant interest due to their fixed-size recurrent state that lowers…
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
Alexander Khazatsky, Karl Pertsch, Suraj Nair +98
The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and robust robotic manipulation policies. Ho…
Language models scale reliably with over-training and on downstream tasks
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar +22
Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps…
Residual Q-Learning: Offline and Online Policy Customization without Value
Chenran Li, Chen Tang, Haruki Nishimura +3
Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complex real-world tasks where handcr…
RAP: Risk-Aware Prediction for Robust Planning
Haruki Nishimura, Jean Mercat, Blake Wulfe +2
Robust planning in interactive scenarios requires predicting the uncertain future to make risk-aware decisions. Unfortunately, due to long-tail safety-critical events, the risk is…
Control-Aware Prediction Objectives for Autonomous Driving
Rowan McAllister, Blake Wulfe, Jean Mercat +3
Autonomous vehicle software is typically structured as a modular pipeline of individual components (e.g., perception, prediction, and planning) to help separate concerns into inter…
Dynamics-Aware Comparison of Learned Reward Functions
Blake Wulfe, Ashwin Balakrishna, Logan Ellis +3
The ability to learn reward functions plays an important role in enabling the deployment of intelligent agents in the real world. However, comparing reward functions, for example a…