What Matters for Adversarial Imitation Learning?
arXiv:2106.00672
Abstract
Adversarial imitation learning has become a popular framework for imitation in continuous control. Over the years, several variations of its components were proposed to enhance the performance of the learned policies as well as the sample complexity of the algorithm. In practice, these choices are rarely tested all together in rigorous empirical studies. It is therefore difficult to discuss and understand what choices, among the high-level algorithmic options as well as low-level implementation details, matter. To tackle this issue, we implement more than 50 of these choices in a generic adversarial imitation learning framework and investigate their impacts in a large-scale study (>500k trained agents) with both synthetic and human-generated demonstrations. While many of our findings confirm common practices, some of them are surprising or even contradict prior work. In particular, our results suggest that artificial demonstrations are not a good proxy for human data and that the very common practice of evaluating imitation algorithms only with synthetic demonstrations may lead to algorithms which perform poorly in the more realistic scenarios with human demonstrations.
References in corpus (24)
- mixup: Beyond Empirical Risk Minimization
- Spectral Normalization for Generative Adversarial Networks
- Trust Region Policy Optimization
- Soft Actor-Critic Algorithms and Applications
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Scaling Laws for Neural Language Models
- Improved Training of Wasserstein GANs
- Dota 2 with Large Scale Deep Reinforcement Learning
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Generative Adversarial Imitation Learning
- Deep Reinforcement Learning that Matters
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Distributed Distributional Deterministic Policy Gradients
- Deep Q-learning from Demonstrations
- Benchmarking Model-Based Reinforcement Learning
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
- What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
- Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
- Acme: A Research Framework for Distributed Reinforcement Learning
- The Mirage of Action-Dependent Baselines in Reinforcement Learning
- Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
- Positive-Unlabeled Reward Learning
- Hyperparameter Selection for Imitation Learning