Imitation Learning as $f$-Divergence Minimization

Abstract

We address the problem of imitation learning with multi-modal demonstrations. Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them. We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorrectly interpolate between such modes. Our key insight is to minimize the right divergence between the learner and the expert state-action distributions, namely the reverse KL divergence or I-projection. We propose a general imitation learning framework for estimating and minimizing any f-Divergence. By plugging in different divergences, we are able to recover existing algorithms such as Behavior Cloning (Kullback-Leibler), GAIL (Jensen Shannon) and Dagger (Total Variation). Empirical results show that our approximate I-projection technique is able to imitate multi-modal behaviors more reliably than GAIL and behavior cloning.

International Workshop on the Algorithmic Foundations of Robotics (WAFR) 2020

Cited by in corpus (28)

• AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
• Primal Wasserstein Imitation Learning
• Imitation Learning via Off-Policy Distribution Matching
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization
• Toward the Fundamental Limits of Imitation Learning
• State Alignment-based Imitation Learning
• Energy-Based Imitation Learning
• $f$-GAIL: Learning $f$-Divergence for Generative Adversarial Imitation Learning
• Non-Adversarial Imitation Learning and its Connections to Adversarial Methods
• Visual Adversarial Imitation Learning using Variational Models
• An operator view of policy gradient methods
• Generative Hybrid Representations for Activity Forecasting with No-Regret Learning
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
• When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence
• MobILE: Model-Based Imitation Learning From Observation Alone
• Imitation Learning with Sinkhorn Distances
• SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching
• LiMIIRL: Lightweight Multiple-Intent Inverse Reinforcement Learning
• Regularized Inverse Reinforcement Learning
• Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Dynamic Regret Convergence Analysis and an Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning
• Mitigation of Adversarial Policy Imitation via Constrained Randomization of Policy (CRoP)
• Imitation with Neural Density Models
• Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap