What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Abstract

Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data. Codebase, datasets, trained models, and more available at https://arise-initiative.github.io/robomimic-web/

arXiv commentCoRL 2021 (Oral)

References in corpus (31)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Deep Reinforcement Learning that Matters
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Reinforcement Learning with Augmented Data
• Benchmarking Model-Based Reinforcement Learning
• robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Interactive Learning from Policy-Dependent Human Feedback
• Critic Regularized Regression
• COMBO: Conservative Offline Model-Based Policy Optimization
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• Benchmarks for Deep Off-Policy Evaluation
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Visual Imitation Made Easy
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity
• Human-in-the-Loop Imitation Learning using Remote Teleoperation
• Hyperparameter Selection for Imitation Learning
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• The MAGICAL Benchmark for Robust Imitation

Cited by in corpus (9)

• Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• A Survey of Embodied Learning for Object-Centric Robotic Manipulation
• The Treachery of Images: Bayesian Scene Keypoints for Deep Policy Learning in Robotic Manipulation
• Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning
• Continuous Control with Action Quantization from Demonstrations
• Curriculum-Based Imitation of Versatile Skills
• Batch Reinforcement Learning from Crowds
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters