Automatic Curricula via Expert Demonstrations
arXiv:2106.09159
Abstract
We propose Automatic Curricula via Expert Demonstrations (ACED), a reinforcement learning (RL) approach that combines the ideas of imitation learning and curriculum learning in order to solve challenging robotic manipulation tasks with sparse reward functions. Curriculum learning solves complicated RL tasks by introducing a sequence of auxiliary tasks with increasing difficulty, yet how to automatically design effective and generalizable curricula remains a challenging research problem. ACED extracts curricula from a small amount of expert demonstration trajectories by dividing demonstrations into sections and initializing training episodes to states sampled from different sections of demonstrations. Through moving the reset states from the end to the beginning of demonstrations as the learning agent improves its performance, ACED not only learns challenging manipulation tasks with unseen initializations and goals, but also discovers novel solutions that are distinct from the demonstrations. In addition, ACED can be naturally combined with other imitation learning methods to utilize expert demonstrations in a more efficient manner, and we show that a combination of ACED with behavior cloning allows pick-and-place tasks to be learned with as few as 1 demonstration and block stacking tasks to be learned with 20 demonstrations.
NeurIPS 2021 Deep Reinforcement Learning Workshop
References in corpus (9)
- Continuous control with deep reinforcement learning
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
- ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst
- Learning Montezuma's Revenge from a Single Demonstration
- Backplay: "Man muss immer umkehren"
- Curiosity-Driven Multi-Criteria Hindsight Experience Replay
- Continual Learning of Control Primitives: Skill Discovery via Reset-Games