Energy-Based Imitation Learning

Abstract

We tackle a common scenario in imitation learning (IL), where agents try to recover the optimal policy from expert demonstrations without further access to the expert or environment reward signals. Except the simple Behavior Cloning (BC) that adopts supervised learning followed by the problem of compounding error, previous solutions like inverse reinforcement learning (IRL) and recent generative adversarial methods involve a bi-level or alternating optimization for updating the reward function and the policy, suffering from high computational cost and training instability. Inspired by recent progress in energy-based model (EBM), in this paper, we propose a simplified IL framework named Energy-Based Imitation Learning (EBIL). Instead of updating the reward and policy iteratively, EBIL breaks out of the traditional IRL paradigm by a simple and flexible two-stage solution: first estimating the expert energy as the surrogate reward function through score matching, then utilizing such a reward for learning the policy by reinforcement learning algorithms. EBIL combines the idea of both EBM and occupancy measure matching, and via theoretic analysis we reveal that EBIL and Max-Entropy IRL (MaxEnt IRL) approaches are two sides of the same coin, and thus EBIL could be an alternative of adversarial IRL methods. Extensive experiments on qualitative and quantitative evaluations indicate that EBIL is able to recover meaningful and interpretative reward signals while achieving effective and comparable performance against existing algorithms on IL benchmarks.

arXiv commentCorrect minor errors. 15 pages (6 pages of supplementary), 8 figures, Accepted by AAMAS-2021

References in corpus (19)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Trust Region Policy Optimization
• Energy-based Generative Adversarial Network
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Reinforcement Learning with Deep Energy-Based Policies
• Generative Moment Matching Networks
• Exploration by Random Network Distillation
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Implicit Generation and Generalization in Energy-Based Models
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Deep Energy Estimator Networks
• Imitation Learning as $f$-Divergence Minimization
• Provably Efficient Imitation Learning from Observation Alone
• State Alignment-based Imitation Learning
• Adversarial Contrastive Estimation
• Neural Empirical Bayes

Cited by in corpus (5)

• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• RITA: Boost Driving Simulators with Realistic Interactive Traffic Flow
• Implicit Behavioral Cloning