Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

Abstract

We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework. The first problem is implicit bias present in the reward functions used in these algorithms. While these biases might work well for some environments, they can also lead to sub-optimal behavior in others. Secondly, even though these algorithms can learn from few expert demonstrations, they require a prohibitively large number of interactions with the environment in order to imitate the expert for many real-world applications. In order to address these issues, we propose a new algorithm called Discriminator-Actor-Critic that uses off-policy Reinforcement Learning to reduce policy-environment interaction sample complexity by an average factor of 10. Furthermore, since our reward function is designed to be unbiased, we can apply our algorithm to many problems without making any task-specific adjustments.

Cited by in corpus (33)

• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Primal Wasserstein Imitation Learning
• Goal-conditioned Imitation Learning
• Imitation Learning via Off-Policy Distribution Matching
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• Error Bounds of Imitating Policies and Environments
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning
• Scalable Bayesian Inverse Reinforcement Learning
• Demonstration-Guided Reinforcement Learning with Learned Skills
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• The MAGICAL Benchmark for Robust Imitation
• Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Critic
• Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Visual Adversarial Imitation Learning using Variational Models
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• Domain-Adversarial and Conditional State Space Model for Imitation Learning
• Imitation Learning with Sinkhorn Distances
• Generative Inverse Deep Reinforcement Learning for Online Recommendation
• Continuous Control with Action Quantization from Demonstrations
• Randomized Adversarial Imitation Learning for Autonomous Driving
• Sample Efficient Social Navigation Using Inverse Reinforcement Learning
• Reward function shape exploration in adversarial imitation learning: an empirical study
• Reparameterized Variational Divergence Minimization for Stable Imitation
• IQ-Learn: Inverse soft-Q Learning for Imitation
• Support-weighted Adversarial Imitation Learning
• Relational Mimic for Visual Adversarial Imitation Learning