103 citations · 251 across the 38 of their papers we have counts for
42 papers
Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency
Brian Zhu, Momen Khalil, E Harrison +17
While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist policies, such as VLAs, poses…
FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space
Michael Murray, Daphne Chen, Simran Bagaria +7
Pretrained generative robot policies based on flow matching and diffusion have achieved impressive results across a wide range of manipulation tasks. Yet real-world deployments rou…
Robot Self-Improvement via Human-Video Dynamics Models
Hanzhi Chen, Anran Zhang, Simon Schaefer +5
A central question in robot learning is how to acquire skills from the kinds of data that humans learn from: passive observation, embodied practice, and the experience of failure.…
From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data
Zhiyuan Feng, Qixiu Li, Huizhi Liang +12
Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision-Language-Action (VLA) models. However, most existing approaches rely on large…
World Model for Robot Learning: A Comprehensive Survey
Bohan Hou, Gen Li, Jindou Jia +15
World models, which are predictive representations of how environments evolve under actions, have become a central component of robot learning. They support policy learning, planni…
SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios
Tian Gao, Celine Tan, Catherine Glossop +8
A fundamental challenge in autonomous driving is the integration of high-level, semantic reasoning for long-tail events with low-level, reactive control for robust driving. While l…
mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs
Jonas Pai, Liam Achenbach, Victoriano Montesinos +3
Prevailing Vision-Language-Action Models (VLAs) for robotic manipulation are built upon vision-language backbones pretrained on large-scale, but disconnected static web data. As a…
Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents
Reuben Tan, Baolin Peng, Zhengyuan Yang +15
Agentic reasoning models trained with multimodal reinforcement learning (MMRL) have become increasingly capable, yet they are almost universally optimized using sparse, outcome-bas…
Multimodal Spatial Language Maps for Robot Navigation and Manipulation
Chenguang Huang, Oier Mees, Andy Zeng +1
Grounding language to a navigating agent's observations can leverage pretrained multimodal foundation models to match perceptions to object or event descriptions. However, previous…
Training Strategies for Efficient Embodied Reasoning
William Chen, Suneel Belkhale, Suvir Mirchandani +4
Robot chain-of-thought reasoning (CoT) -- wherein a model predicts helpful intermediate representations before choosing actions -- provides an effective method for improving the ge…
Towards Embodiment Scaling Laws in Robot Locomotion
Bo Ai, Liu Dai, Nico Bohlinger +7
Cross-embodiment generalization underpins the vision of building generalist embodied agents for any robot, yet its enabling factors remain poorly understood. We investigate embodim…
FAST: Efficient Action Tokenization for Vision-Language-Action Models
Karl Pertsch, Kyle Stachowicz, Brian Ichter +6
Autoregressive sequence models, such as Transformer-based vision-language action (VLA) policies, can be tremendously effective for capturing complex and generalizable robotic behav…
Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding
Joshua Jones, Oier Mees, Carmelo Sferrazza +3
Interacting with the world is a multi-sensory experience: achieving effective general-purpose interaction requires making use of all available modalities -- including vision, touch…
RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning
Charles Xu, Qiyang Li, Jianlan Luo +1
Recent advances in robotic foundation models have enabled the development of generalist policies that can adapt to diverse tasks. While these models show impressive flexibility, th…
GHIL-Glue: Hierarchical Control with Filtered Subgoal Images
Kyle B. Hatch, Ashwin Balakrishna, Oier Mees +8
Image and video generative models that are pre-trained on Internet-scale data can greatly increase the generalization capacity of robot learning systems. These models can function…
Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models
Nils Blank, Moritz Reuss, Marcel Rühle +4
A central challenge towards developing robots that can relate human language to their perception and actions is the scarcity of natural language annotations in diverse robot datase…
The Ingredients for Robotic Diffusion Transformers
Sudeep Dasari, Oier Mees, Sebastian Zhao +2
In recent years roboticists have achieved remarkable progress in solving increasingly general tasks on dexterous robotic hardware by leveraging high capacity Transformer network ar…
LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Videos
Noriaki Hirose, Catherine Glossop, Ajay Sridhar +3
The world is filled with a wide variety of objects. For robots to be useful, they need the ability to find arbitrary objects described by people. In this paper, we present LeLaN(Le…
Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar +1
Large, general-purpose robotic policies trained on diverse demonstration datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of…
Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation
Vivek Myers, Bill Chunyuan Zheng, Oier Mees +2
Learned language-conditioned robot policies often struggle to effectively adapt to new real-world tasks even when pre-trained across a diverse set of instructions. We propose a nov…
Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation
Ria Doshi, Homer Walke, Oier Mees +2
Modern machine learning systems rely on large datasets to attain broad generalization, and this often poses a challenge in robot learning, where each robotic platform and task migh…
Autonomous Improvement of Instruction Following Skills via Foundation Models
Zhiyuan Zhou, Pranav Atreya, Abraham Lee +3
Intelligent instruction-following robots capable of improving from autonomously collected experience have the potential to transform robot learning: instead of collecting costly te…
Robotic Control via Embodied Chain-of-Thought Reasoning
Michał Zawalski, William Chen, Karl Pertsch +3
A key limitation of learned robot control policies is their inability to generalize outside their training data. Recent works on vision-language-action models (VLAs) have shown tha…
Octo: An Open-Source Generalist Robot Policy
Octo Model Team, Dibya Ghosh, Homer Walke +16
Large policies pretrained on diverse robot datasets have the potential to transform robotic learning: instead of training new policies from scratch, such generalist robot policies…