A Pragmatic Look at Deep Imitation Learning

Abstract

The introduction of the generative adversarial imitation learning (GAIL) algorithm has spurred the development of scalable imitation learning approaches using deep neural networks. Many of the algorithms that followed used a similar procedure, combining on-policy actor-critic algorithms with inverse reinforcement learning. More recently there have been an even larger breadth of approaches, most of which use off-policy algorithms. However, with the breadth of algorithms, everything from datasets to base reinforcement learning algorithms to evaluation settings can vary, making it difficult to fairly compare them. In this work we re-implement 6 different IL algorithms, updating 3 of them to be off-policy, base them on a common off-policy algorithm (SAC), and evaluate them on a widely-used expert trajectory dataset (D4RL) for the most common benchmark (MuJoCo). After giving all algorithms the same hyperparameter optimisation budget, we compare their results for a range of expert trajectories. In summary, GAIL, with all of its improvements, consistently performs well across a range of sample sizes, AdRIL is a simple contender that performs well with one important hyperparameter to tune, and behavioural cloning remains a strong baseline when data is more plentiful.

Asian Conference on Machine Learning, 2023

References in corpus (11)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Generative Moment Matching Networks
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
• Training generative neural networks via Maximum Mean Discrepancy optimization
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation
• Hyperparameter Selection for Imitation Learning
• Addressing reward bias in Adversarial Imitation Learning with neutral reward functions