Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies

Abstract

Policy gradient (PG) methods are a widely used reinforcement learning methodology in many applications such as video games, autonomous driving, and robotics. In spite of its empirical success, a rigorous understanding of the global convergence of PG methods is lacking in the literature. In this work, we close the gap by viewing PG methods from a nonconvex optimization perspective. In particular, we propose a new variant of PG methods for infinite-horizon problems that uses a random rollout horizon for the Monte-Carlo estimation of the policy gradient. This method then yields an unbiased estimate of the policy gradient with bounded variance, which enables the tools from nonconvex optimization to be applied to establish global convergence. Employing this perspective, we first recover the convergence results with rates to the stationary-point policies in the literature. More interestingly, motivated by advances in nonconvex optimization, we modify the proposed PG method by introducing periodically enlarged stepsizes. The modified algorithm is shown to escape saddle points under mild assumptions on the reward and the policy parameterization. Under a further strict saddle points assumption, this result establishes convergence to essentially locally-optimal policies of the underlying problem, and thus bridges the gap in existing literature on the convergence of PG methods. Results from experiments on the inverted pendulum are then provided to corroborate our theory, namely, by slightly reshaping the reward function to satisfy our assumption, unfavorable saddle points can be avoided and better limit points can be attained. Intriguingly, this empirical finding justifies the benefit of reward-reshaping from a nonconvex optimization perspective.

arXiv commentInitially submitted in Jan. 2019. Accepted to SIAM Journal on Control and Optimization (SICON)

References in corpus (13)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Experiments with Infinite-Horizon, Policy-Gradient Estimation
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• Escaping Saddles with Stochastic Gradients
• LQR through the Lens of First Order Methods: Discrete-time Case
• Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
• Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
• Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent

Cited by in corpus (21)

• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
• Variational Policy Gradient Method for Reinforcement Learning with General Utilities
• A Finite Time Analysis of Two Time-Scale Actor Critic Methods
• Non-asymptotic Convergence Analysis of Two Time-scale (Natural) Actor-Critic Algorithms
• Improving Sample Complexity Bounds for (Natural) Actor-Critic Algorithms
• On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
• Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithm
• On the Analysis of Model-free Methods for the Linear Quadratic Regulator
• Zeroth-order Deterministic Policy Gradient
• Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate
• When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence
• Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents
• Bregman Gradient Policy Optimization
• Some Limit Properties of Markov Chains Induced by Stochastic Recursive Algorithms
• A Study of Policy Gradient on a Class of Exactly Solvable Models
• Statistical Learning Based Joint Antenna Selection and User Scheduling for Single-Cell Massive MIMO Systems
• Parameter Critic: a Model Free Variance Reduction Method Through Imperishable Samples