VILD: Variational Imitation Learning with Diverse-quality Demonstrations

Abstract

The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose a new IL method called \underline{v}ariational \underline{i}mitation \underline{l}earning with \underline{d}iverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators' expertise with a probabilistic graphical model and estimate it along with a reward function. We show that a naive approach to estimation is not suitable to large state and action spaces, and fix its issues by using a variational approach which can be easily implemented using existing reinforcement learning methods. Experiments on continuous-control benchmarks demonstrate that VILD outperforms state-of-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.

References in corpus (11)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• A Closer Look at Memorization in Deep Networks
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Learning From Noisy Singly-labeled Data
• Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
• Maximum Entropy Semi-Supervised Inverse Reinforcement Learning
• Generative predecessor models for sample-efficient imitation learning