Imitation Learning from Imperfect Demonstration

Abstract

Imitation learning (IL) aims to learn an optimal policy from demonstrations. However, such demonstrations are often imperfect since collecting optimal ones is costly. To effectively learn from imperfect demonstrations, we propose a novel approach that utilizes confidence scores, which describe the quality of demonstrations. More specifically, we propose two confidence-based IL methods, namely two-step importance weighting IL (2IWIL) and generative adversarial IL with imperfect demonstration and confidence (IC-GAIL). We show that confidence scores given only to a small portion of sub-optimal demonstrations significantly improve the performance of IL both theoretically and empirically.

References in corpus (4)

• Trust Region Policy Optimization
• A Deep Reinforcement Learning Chatbot
• Composite Binary Losses
• Avoiding Wireheading with Value Reinforcement Learning

Cited by in corpus (7)

• Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations
• Reinforcement Learning with Supervision from Noisy Demonstrations
• Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• No Need for Interactions: Robust Model-Based Imitation Learning using Neural ODE
• A Symmetric Loss Perspective of Reliable Machine Learning
• Towards intervention-centric causal reasoning in learning agents