Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate
arXiv:2003.03709
Abstract
Generative adversarial imitation learning (GAIL) demonstrates tremendous success in practice, especially when combined with neural networks. Different from reinforcement learning, GAIL learns both policy and reward function from expert (human) demonstration. Despite its empirical success, it remains unclear whether GAIL with neural networks converges to the globally optimal solution. The major difficulty comes from the nonconvex-nonconcave minimax optimization structure. To bridge the gap between practice and theory, we analyze a gradient-based algorithm with alternating updates and establish its sublinear convergence to the globally optimal solution. To the best of our knowledge, our analysis establishes the global optimality and convergence rate of GAIL with neural networks for the first time.
arXiv comment42 pages; accepted to ICML; initial draft submitted in Feb, 2020
References in corpus (13)
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- Learning human behaviors from motion capture by adversarial imitation
- On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
- Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
- Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
- Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
- Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
- Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies
- An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient
- On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator
- Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima