Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs
arXiv:2104.04901
Abstract
While the techniques in optimal control theory are often model-based, the policy optimization (PO) approach directly optimizes the performance metric of interest. Even though it has been an essential approach for reinforcement learning problems, there is little theoretical understanding on its performance. In this paper, we focus on the risk-constrained linear quadratic regulator (RC-LQR) problem via the PO approach, which requires addressing a challenging non-convex constrained optimization problem. To solve it, we first build on our earlier result that an optimal policy has a time-invariant affine structure to show that the associated Lagrangian function is coercive, locally gradient dominated and has local Lipschitz continuous gradient, based on which we establish strong duality. Then, we design policy gradient primal-dual methods with global convergence guarantees in both model-based and sample-based settings. Finally, we use samples of system trajectories in simulations to validate our methods.
References in corpus (9)
- Constrained Policy Optimization
- LQR through the Lens of First Order Methods: Discrete-time Case
- Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
- Learning the Globally Optimal Distributed LQ Regulator
- Convergent Policy Optimization for Safe Reinforcement Learning
- Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games
- Risk Averse Robust Adversarial Reinforcement Learning
- Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulator
- Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Cost