Primal-dual Learning for the Model-free Risk-constrained Linear Quadratic Regulator
arXiv:2011.10931
Abstract
Risk-aware control, though with promise to tackle unexpected events, requires a known exact dynamical model. In this work, we propose a model-free framework to learn a risk-aware controller with a focus on the linear system. We formulate it as a discrete-time infinite-horizon LQR problem with a state predictive variance constraint. To solve it, we parameterize the policy with a feedback gain pair and leverage primal-dual methods to optimize it by solely using data. We first study the optimization landscape of the Lagrangian function and establish the strong duality in spite of its non-convex nature. Alongside, we find that the Lagrangian function enjoys an important local gradient dominance property, which is then exploited to develop a convergent random search algorithm to learn the dual function. Furthermore, we propose a primal-dual algorithm with global convergence to learn the optimal policy-multiplier pair. Finally, we validate our results via simulations.
arXiv commentTo appear in the Annual Conference on Learning for Dynamics and Control (L4DC) 2021
References in corpus (10)
- Continuous control with deep reinforcement learning
- Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
- Reward Constrained Policy Optimization
- Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems
- LQR through the Lens of First Order Methods: Discrete-time Case
- Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
- Learning the Globally Optimal Distributed LQ Regulator
- Convergent Policy Optimization for Safe Reinforcement Learning
- Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs
- Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Cost