Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
arXiv:2211.04786
Abstract
Deep Reinforcement Learning has been successfully applied to learn robotic control. However, the corresponding algorithms struggle when applied to problems where the agent is only rewarded after achieving a complex task. In this context, using demonstrations can significantly speed up the learning process, but demonstrations can be costly to acquire. In this paper, we propose to leverage a sequential bias to learn control policies for complex robotic tasks using a single demonstration. To do so, our method learns a goal-conditioned policy to control a system between successive low-dimensional goals. This sequential goal-reaching approach raises a problem of compatibility between successive goals: we need to ensure that the state resulting from reaching a goal is compatible with the achievement of the following goals. To tackle this problem, we present a new algorithm called DCIL-II. We show that DCIL-II can solve with unprecedented sample efficiency some challenging simulated tasks such as humanoid locomotion and stand-up as well as fast running with a simulated Cassie robot. Our method leveraging sequentiality is a step towards the resolution of complex robotic tasks under minimal specification effort, a key feature for the next generation of autonomous robots.
References in corpus (13)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Solving Rubik's Cube with a Robot Hand
- OpenAI Gym
- VIME: Variational Information Maximizing Exploration
- Visual Reinforcement Learning with Imagined Goals
- Go-Explore: a New Approach for Hard-Exploration Problems
- Learning Montezuma's Revenge from a Single Demonstration
- Learning Multi-Level Hierarchies with Hindsight
- Hierarchical visuomotor control of humanoids
- Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
- Primal Wasserstein Imitation Learning
- Learnings Options End-to-End for Continuous Action Tasks
- Goal-Conditioned Reinforcement Learning with Imagined Subgoals