$f$-GAIL: Learning $f$-Divergence for Generative Adversarial Imitation Learning

Abstract

Imitation learning (IL) aims to learn a policy from expert demonstrations that minimizes the discrepancy between the learner and expert behaviors. Various imitation learning algorithms have been proposed with different pre-determined divergences to quantify the discrepancy. This naturally gives rise to the following question: Given a set of expert demonstrations, which divergence can recover the expert policy more accurately with higher data efficiency? In this work, we propose $f$-GAIL, a new generative adversarial imitation learning (GAIL) model, that automatically learns a discrepancy measure from the $f$-divergence family as well as a policy capable of producing expert-like behaviors. Compared with IL baselines with various predefined divergence measures, $f$-GAIL learns better policies with higher data efficiency in six physics-based control tasks.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Input Convex Neural Networks
• Imitation Learning as $f$-Divergence Minimization
• A Divergence Minimization Perspective on Imitation Learning Methods
• Training Deep Energy-Based Models with f-Divergence Minimization
• Augmenting GAIL with BC for sample efficient imitation learning
• Reparameterized Variational Divergence Minimization for Stable Imitation

Cited by in corpus (2)

• PAIL: Performance based Adversarial Imitation Learning Engine for Carbon Neutral Optimization
• Inverse Contextual Bandits: Learning How Behavior Evolves over Time