Sample-Efficient Imitation Learning via Generative Adversarial Nets

Abstract

GAIL is a recent successful imitation learning architecture that exploits the adversarial training procedure introduced in GANs. Albeit successful at generating behaviours similar to those demonstrated to the agent, GAIL suffers from a high sample complexity in the number of interactions it has to carry out in the environment in order to achieve satisfactory performance. We dramatically shrink the amount of interactions with the environment necessary to learn well-behaved imitation policies, by up to several orders of magnitude. Our framework, operating in the model-free regime, exhibits a significant increase in sample-efficiency over previous methods by simultaneously a) learning a self-tuned adversarially-trained surrogate reward and b) leveraging an off-policy actor-critic architecture. We show that our approach is simple to implement and that the learned agents remain remarkably stable, as shown in our experiments that span a variety of continuous control tasks. Video visualisations available at: \url{https://youtu.be/-nCsqUJnRKU}.

arXiv commentPublished as a conference paper for AISTATS 2019

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Trust Region Policy Optimization
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Noisy Networks for Exploration
• Layer Normalization
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Sample Efficient Actor-Critic with Experience Replay
• Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• Learning values across many orders of magnitude
• Are GANs Created Equal? A Large-Scale Study
• Model-Free Imitation Learning with Policy Optimization

Cited by in corpus (5)

• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• State Alignment-based Imitation Learning
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Positive-Unlabeled Reward Learning
• Model-Based Imitation Learning Using Entropy Regularization of Model and Policy