Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations

Abstract

Model-free deep reinforcement learning (RL) has demonstrated its superiority on many complex sequential decision-making problems. However, heavy dependence on dense rewards and high sample-complexity impedes the wide adoption of these methods in real-world scenarios. On the other hand, imitation learning (IL) learns effectively in sparse-rewarded tasks by leveraging the existing expert demonstrations. In practice, collecting a sufficient amount of expert demonstrations can be prohibitively expensive, and the quality of demonstrations typically limits the performance of the learning policy. In this work, we propose Self-Adaptive Imitation Learning (SAIL) that can achieve (near) optimal performance given only a limited number of sub-optimal demonstrations for highly challenging sparse reward tasks. SAIL bridges the advantages of IL and RL to reduce the sample complexity substantially, by effectively exploiting sup-optimal demonstrations and efficiently exploring the environment to surpass the demonstrated performance. Extensive empirical results show that not only does SAIL significantly improve the sample-efficiency but also leads to much better final performance across different continuous control tasks, comparing to the state-of-the-art.

References in corpus (11)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Q-learning from Demonstrations
• Exploration by Random Network Distillation
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
• Imitation Learning via Off-Policy Distribution Matching
• Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation

Cited by in corpus (3)

• Transfer Learning in Deep Reinforcement Learning: A Survey
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Learning from Imperfect Demonstrations from Agents with Varying Dynamics