Imitation Learning via Off-Policy Distribution Matching

Abstract

When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly data-inefficient or alter the original objective in a manner that can drastically change its optimum. In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective. In addition to the data-efficiency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary.Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, finding that it can achieve state-of-the-art sample efficiency and performance.

References in corpus (6)

• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• End to End Learning for Self-Driving Cars
• Soft Actor-Critic Algorithms and Applications
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Imitation Learning as $f$-Divergence Minimization
• Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

Cited by in corpus (16)

• Error Bounds of Imitating Policies and Environments
• Scalable Bayesian Inverse Reinforcement Learning
• Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• Non-Adversarial Imitation Learning and its Connections to Adversarial Methods
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Visual Adversarial Imitation Learning using Variational Models
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• Inverse Decision Modeling: Learning Interpretable Representations of Behavior
• Implicit Behavioral Cloning
• Imitation Learning with Sinkhorn Distances
• IQ-Learn: Inverse soft-Q Learning for Imitation
• Imitation with Neural Density Models
• RAIL: A modular framework for Reinforcement-learning-based Adversarial Imitation Learning