Demonstration-Guided Reinforcement Learning with Learned Skills

Abstract

Demonstration-guided reinforcement learning (RL) is a promising approach for learning complex behaviors by leveraging both reward feedback and a set of target task demonstrations. Prior approaches for demonstration-guided RL treat every new task as an independent learning problem and attempt to follow the provided demonstrations step-by-step, akin to a human trying to imitate a completely unseen behavior by following the demonstrator's exact muscle movements. Naturally, such learning will be slow, but often new behaviors are not completely unseen: they share subtasks with behaviors we have previously learned. In this work, we aim to exploit this shared subtask structure to increase the efficiency of demonstration-guided RL. We first learn a set of reusable skills from large offline datasets of prior experience collected across many tasks. We then propose Skill-based Learning with Demonstrations (SkiLD), an algorithm for demonstration-guided RL that efficiently leverages the provided demonstrations by following the demonstrated skills instead of the primitive actions, resulting in substantial performance improvements over prior demonstration-guided RL approaches. We validate the effectiveness of our approach on long-horizon maze navigation and complex robot manipulation tasks.

References in corpus (14)

• Soft Actor-Critic Algorithms and Applications
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Deep Q-learning from Demonstrations
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Learning human behaviors from motion capture by adversarial imitation
• Neural probabilistic motor primitives for humanoid control
• Learning Multi-Level Hierarchies with Hindsight
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning