MobILE: Model-Based Imitation Learning From Observation Alone
arXiv:2102.10769
Abstract
This paper studies Imitation Learning from Observations alone (ILFO) where the learner is presented with expert demonstrations that consist only of states visited by an expert (without access to actions taken by the expert). We present a provably efficient model-based framework MobILE to solve the ILFO problem. MobILE involves carefully trading off strategic exploration against imitation - this is achieved by integrating the idea of optimism in the face of uncertainty into the distribution matching imitation learning (IL) framework. We provide a unified analysis for MobILE, and demonstrate that MobILE enjoys strong performance guarantees for classes of MDP dynamics that satisfy certain well studied notions of structural complexity. We also show that the ILFO problem is strictly harder than the standard IL problem by presenting an exponential sample complexity separation between IL and ILFO. We complement these theoretical results with experimental simulations on benchmark OpenAI Gym tasks that indicate the efficacy of MobILE. Code for implementing the MobILE framework is available at https://github.com/rahulkidambi/MobILE-NeurIPS2021.
arXiv comment29 pages, 7 figures
References in corpus (28)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- End-to-End Training of Deep Visuomotor Policies
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Generative Adversarial Imitation Learning
- Continuous Deep Q-Learning with Model-based Acceleration
- Model-Ensemble Trust-Region Policy Optimization
- Benchmarking Model-Based Reinforcement Learning
- MOReL : Model-Based Offline Reinforcement Learning
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- Reinforcement and Imitation Learning via Interactive No-Regret Learning
- When to Trust Your Model: Model-Based Policy Optimization
- Randomized Prior Functions for Deep Reinforcement Learning
- Generative Adversarial Imitation from Observation
- Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
- A Game Theoretic Framework for Model Based Reinforcement Learning
- Imitating Latent Policies from Observation
- Model-based Reinforcement Learning and the Eluder Dimension
- Imitation Learning as -Divergence Minimization
- Active Learning for Nonlinear System Identification with Guarantees
- Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
- Information Theoretic Regret Bounds for Online Nonlinear Control
- Provably Efficient Imitation Learning from Observation Alone
- Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
- Convergence of Value Aggregation for Imitation Learning
- Off-Policy Imitation Learning from Observations
- Reinforcement Learning with Videos: Combining Offline Observations with Interaction
- Provably Efficient Model-based Policy Adaptation