Provably Efficient Imitation Learning from Observation Alone

Abstract

We study Imitation Learning (IL) from Observations alone (ILFO) in large-scale MDPs. While most IL algorithms rely on an expert to directly provide actions to the learner, in this setting the expert only supplies sequences of observations. We design a new model-free algorithm for ILFO, Forward Adversarial Imitation Learning (FAIL), which learns a sequence of time-dependent policies by minimizing an Integral Probability Metric between the observation distributions of the expert policy and the learner. FAIL is the first provably efficient algorithm in ILFO setting, which learns a near-optimal policy with a number of samples that is polynomial in all relevant parameters but independent of the number of unique observations. The resulting theory extends the domain of provably sample efficient learning algorithms beyond existing results, which typically only consider tabular reinforcement learning settings or settings that require access to a near-optimal reset distribution. We also investigate the extension of FAIL in a model-based setting. Finally we demonstrate the efficacy of FAIL on multiple OpenAI Gym control tasks.

ICML 2019

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Trust Region Policy Optimization
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Learning to Search Better Than Your Teacher
• Imitating Latent Policies from Observation
• Learning to Filter with Predictive State Inference Machines
• Learning to Search for Dependencies

Cited by in corpus (18)

• State Alignment-based Imitation Learning
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Constrained episodic reinforcement learning in concave-convex and knapsack settings
• Energy-Based Imitation Learning
• Imitation Learning: Progress, Taxonomies and Challenges
• XIRL: Cross-embodiment Inverse Reinforcement Learning
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Robust Imitation Learning from Noisy Demonstrations
• A Simple Reward-free Approach to Constrained Reinforcement Learning
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Provable Representation Learning for Imitation Learning via Bi-level Optimization
• MobILE: Model-Based Imitation Learning From Observation Alone
• Interactive Learning from Activity Description
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Imitation Learning by Reinforcement Learning
• Towards Learning to Imitate from a Single Video Demonstration