2.3k citations · 3.8k across the 166 of their papers we have counts for
27 papers · 1 filter
What's the Move? Hybrid Imitation Learning via Salient Points
Priya Sundaresan, Hengyuan Hu, Quan Vuong +2
While imitation learning (IL) offers a promising framework for teaching robots various behaviors, learning complex tasks remains challenging. Existing IL policies struggle to gener…
Vision Language Models are In-Context Value Learners
Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid +15
Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estimator, or temporal…
RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation
Soroush Nasiriany, Sean Kirmani, Tianli Ding +5
We explore how intermediate policy representations can facilitate generalization by providing guidance on how to perform manipulation tasks. Existing representations such as langua…
Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration
Jennifer Grannen, Siddharth Karamcheti, Suvir Mirchandani +2
We introduce Vocal Sandbox, a framework for enabling seamless human-robot collaboration in situated environments. Systems in our framework are characterized by their ability to ada…
So You Think You Can Scale Up Autonomous Robot Data Collection?
Suvir Mirchandani, Suneel Belkhale, Joey Hejna +3
A long-standing goal in robot learning is to develop methods for robots to acquire new skills autonomously. While reinforcement learning (RL) comes with the promise of enabling aut…
RoboCrowd: Scaling Robot Data Collection through Crowdsourcing
Suvir Mirchandani, David D. Yuan, Kaylee Burns +4
In recent years, imitation learning from large-scale human demonstrations has emerged as a promising paradigm for training robot policies. However, the burden of collecting large q…
Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation
Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta +7
How can robot manipulation policies generalize to novel tasks involving unseen object types and new motions? In this paper, we provide a solution in terms of predicting motion info…
MotIF: Motion Instruction Fine-tuning
Minyoung Hwang, Joey Hejna, Dorsa Sadigh +1
While success in many robotics tasks can be determined by only observing the final state and how it differs from the initial state - e.g., if an apple is picked up - many tasks req…
Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning
Joey Hejna, Chethan Bhateja, Yichen Jiang +2
Increasingly large imitation learning datasets are being collected with the goal of training foundation models for robotics. However, despite the fact that data selection has been…
FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learning
Li-Heng Lin, Yuchen Cui, Amber Xie +2
Few-shot imitation learning relies on only a small amount of task-specific demonstrations to efficiently adapt a policy for a given downstream tasks. Retrieval-based methods come w…
FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishes
Rajat Kumar Jenamani, Priya Sundaresan, Maram Sakr +2
Robot-assisted feeding has the potential to improve the quality of life for individuals with mobility limitations who are unable to feed themselves independently. However, there ex…
OpenVLA: An Open-Source Vision-Language-Action Model
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti +15
Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rath…
VADER: Visual Affordance Detection and Error Recovery for Multi Robot Human Collaboration
Michael Ahn, Montserrat Gonzalez Arenas, Matthew Bennice +22
Robots today can exploit the rich world knowledge of large language models to chain simple behavioral skills into long-horizon tasks. However, robots often get interrupted during l…
Octo: An Open-Source Generalist Robot Policy
Octo Model Team, Dibya Ghosh, Homer Walke +16
Large policies pretrained on diverse robot datasets have the potential to transform robotic learning: instead of training new policies from scratch, such generalist robot policies…
Policy Learning with a Language Bottleneck
Megha Srivastava, Cedric Colas, Dorsa Sadigh +1
Modern AI systems such as self-driving cars and game-playing agents can achieve superhuman performance, but often lack human-like generalization, interpretability, and inter-operab…
RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
Priya Sundaresan, Quan Vuong, Jiayuan Gu +10
Natural language and images are commonly used as goal representations in goal-conditioned imitation learning (IL). However, natural language can be ambiguous and images can be over…
Explore until Confident: Efficient Exploration for Embodied Question Answering
Allen Z. Ren, Jaden Clark, Anushri Dixit +3
We consider the problem of Embodied Question Answering (EQA), which refers to settings where an embodied agent such as a robot needs to actively explore an environment to gather in…
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
Alexander Khazatsky, Karl Pertsch, Suraj Nair +98
The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and robust robotic manipulation policies. Ho…
Efficient Data Collection for Robotic Manipulation via Compositional Generalization
Jensen Gao, Annie Xie, Ted Xiao +2
Data collection has become an increasingly important problem in robotic manipulation, yet there still lacks much understanding of how to effectively collect data to facilitate broa…
RT-H: Action Hierarchies Using Language
Suneel Belkhale, Tianli Ding, Ted Xiao +6
Language provides a way to break down complex concepts into digestible pieces. Recent works in robot imitation learning use language-conditioned policies that predict actions given…
Pushing the Limits of Cross-Embodiment Learning for Manipulation and Navigation
Jonathan Yang, Catherine Glossop, Arjun Bhorkar +5
Recent years in robotics and imitation learning have shown remarkable progress in training large-scale foundation models by leveraging data across a multitude of embodiments. The s…
Batch Active Learning of Reward Functions from Human Preferences
Erdem Bıyık, Nima Anari, Dorsa Sadigh
Data generation and labeling are often expensive in robot learning. Preference-based learning is a concept that enables reliable labeling by querying users with preference question…
Learning to Learn Faster from Human Feedback with Language Model Predictive Control
Jacky Liang, Fei Xia, Wenhao Yu +47
Large language models (LLMs) have been shown to exhibit a wide range of capabilities, such as writing robot code from language commands -- enabling non-experts to direct robot beha…
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna +3
Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning; adoption that has fu…