Error Bounds of Imitating Policies and Environments

Abstract

Imitation learning trains a policy by mimicking expert demonstrations. Various imitation methods were proposed and empirically evaluated, meanwhile, their theoretical understanding needs further studies. In this paper, we firstly analyze the value gap between the expert policy and imitated policies by two imitation methods, behavioral cloning and generative adversarial imitation. The results support that generative adversarial imitation can reduce the compounding errors compared to behavioral cloning, and thus has a better sample complexity. Noticed that by considering the environment transition model as a dual agent, imitation learning can also be used to learn the environment model. Therefore, based on the bounds of imitating policies, we further analyze the performance of imitating environments. The results show that environment models can be more effectively imitated by generative adversarial imitation than behavioral cloning, suggesting a novel application of adversarial imitation for model-based reinforcement learning. We hope these results could inspire future advances in imitation learning and model-based reinforcement learning.

arXiv commentAppears in NeurIPS 2020

References in corpus (13)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Generative Adversarial Imitation Learning
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Spectrally-normalized margin bounds for neural networks
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• When to Trust Your Model: Model-Based Policy Optimization
• Learning to Search Better Than Your Teacher
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• On the Discrimination-Generalization Tradeoff in GANs
• Imitation Learning via Off-Policy Distribution Matching
• On Computation and Generalization of Generative Adversarial Imitation Learning
• Imitation Learning from Pixel-Level Demonstrations by HashReward

Cited by in corpus (3)

• Adversarial Constrained Bidding via Minimax Regret Optimization with Causality-Aware Reinforcement Learning
• RITA: Boost Driving Simulators with Realistic Interactive Traffic Flow
• Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally