Model-Free Imitation Learning with Policy Optimization

Abstract

In imitation learning, an agent learns how to behave in an environment with an unknown cost function by mimicking expert demonstrations. Existing imitation learning algorithms typically involve solving a sequence of planning or reinforcement learning problems. Such algorithms are therefore not directly applicable to large, high-dimensional environments, and their performance can significantly degrade if the planning problems are not solved to optimality. Under the apprenticeship learning formalism, we develop alternative model-free algorithms for finding a parameterized stochastic policy that performs at least as well as an expert policy on an unknown cost function, based on sample trajectories from the expert. Our approach, based on policy gradients, scales to large continuous environments with guaranteed convergence to local minima.

In Proceedings of the 33rd International Conference on Machine Learning, 2016

Cited by in corpus (18)

• Deep Reinforcement Learning: An Overview
• Generative Adversarial Imitation Learning
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Modeling Human Driving Behavior through Generative Adversarial Imitation Learning
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Paraphrase Generation with Deep Reinforcement Learning
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Self-Supervised Reinforcement Learning for Recommender Systems
• A New Approach for Resource Scheduling with Deep Reinforcement Learning
• Hindsight Trust Region Policy Optimization
• Visual Forecasting by Imitating Dynamics in Natural Sequences
• Imitating Driver Behavior with Generative Adversarial Networks
• Inverse Decision Modeling: Learning Interpretable Representations of Behavior
• On Value Discrepancy of Imitation Learning
• Learning to Optimize via Wasserstein Deep Inverse Optimal Control
• Relational Mimic for Visual Adversarial Imitation Learning