f-IRL: Inverse Reinforcement Learning via State Marginal Matching

Abstract

Imitation learning is well-suited for robotic tasks where it is difficult to directly program the behavior or specify a cost for optimal control. In this work, we propose a method for learning the reward function (and the corresponding policy) to match the expert state density. Our main result is the analytic gradient of any f-divergence between the agent and expert state distribution w.r.t. reward parameters. Based on the derived gradient, we present an algorithm, f-IRL, that recovers a stationary reward function from the expert density by gradient descent. We show that f-IRL can learn behaviors from a hand-designed target state density or implicitly through expert observations. Our method outperforms adversarial imitation learning methods in terms of sample efficiency and the required number of expert trajectories on IRL benchmarks. Moreover, we show that the recovered reward function can be used to quickly solve downstream tasks, and empirically demonstrate its utility on hard-to-explore tasks and for behavior transfer across changes in dynamics.

arXiv commentThe first four authors have equal contribution (orders determined by dice rolling), and the last two authors have equal advising. The paper is accepted by Conference on Robot Learning (CoRL) 2020. Project videos and code link are available at https://sites.google.com/view/f-irl/home

References in corpus (16)

• Estimating Mutual Information
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Learning agile and dynamic motor skills for legged robots
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Deep Reinforcement Learning that Matters
• Maximum Entropy Deep Inverse Reinforcement Learning
• Safe Exploration in Continuous Action Spaces
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Efficient Exploration via State Marginal Matching
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Imitation Learning as $f$-Divergence Minimization
• Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimation
• A Divergence Minimization Perspective on Imitation Learning Methods
• State Alignment-based Imitation Learning
• Energy-Based Imitation Learning