A Divergence Minimization Perspective on Imitation Learning Methods

Abstract

In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations. The most common approaches under this Imitation Learning (IL) framework are Behavioural Cloning (BC), and Inverse Reinforcement Learning (IRL). Recent methods for IRL have demonstrated the capacity to learn effective policies with access to a very limited set of demonstrations, a scenario in which BC methods often fail. Unfortunately, due to multiple factors of variation, directly comparing these methods does not provide adequate intuition for understanding this difference in performance. In this work, we present a unified probabilistic perspective on IL algorithms based on divergence minimization. We present $f$-MAX, an $f$-divergence generalization of AIRL [Fu et al., 2018], a state-of-the-art IRL method. $f$-MAX enables us to relate prior IRL methods such as GAIL [Ho & Ermon, 2016] and AIRL [Fu et al., 2018], and understand their algorithmic properties. Through the lens of divergence minimization we tease apart the differences between BC and successful IRL approaches, and empirically evaluate these nuances on simulated high-dimensional continuous control domains. Our findings conclusively identify that IRL's state-marginal matching objective contributes most to its superior performance. Lastly, we apply our new understanding of IL methods to the problem of state-marginal matching, where we demonstrate that in simulated arm pushing environments we can teach agents a diverse range of behaviours using simply hand-specified state distributions and no reward functions or expert demonstrations. For datasets and reproducing results please refer to https://github.com/KamyarGh/rl_swiss/blob/master/reproducing/fmax_paper.md .

Published at Conference on Robot Learning (CoRL) 2019. For datasets and reproducing results please refer to https://github.com/KamyarGh/rl_swiss/blob/master/reproducing/fmax_paper.md

References in corpus (12)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Improved Training of Wasserstein GANs
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Optimal control as a graphical model inference problem
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Efficient Exploration via State Marginal Matching
• Reward Augmented Maximum Likelihood for Neural Structured Prediction
• Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
• Formal Limitations on the Measurement of Mutual Information

Cited by in corpus (7)

• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Action and Perception as Divergence Minimization
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• $f$-GAIL: Learning $f$-Divergence for Generative Adversarial Imitation Learning
• Generative Hybrid Representations for Activity Forecasting with No-Regret Learning
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Reparameterized Variational Divergence Minimization for Stable Imitation