Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with $\sqrt{T}$ Regret

Abstract

We consider the task of learning to control a linear dynamical system under fixed quadratic costs, known as the Linear Quadratic Regulator (LQR) problem. While model-free approaches are often favorable in practice, thus far only model-based methods, which rely on costly system identification, have been shown to achieve regret that scales with the optimal dependence on the time horizon T. We present the first model-free algorithm that achieves similar regret guarantees. Our method relies on an efficient policy gradient scheme, and a novel and tighter analysis of the cost of exploration in policy space in this setting.

References in corpus (8)

• Continuous control with deep reinforcement learning
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems
• Logarithmic Regret for Online Control
• Model-Free Linear Quadratic Control via Reduction to Expert Prediction
• Black-Box Control for Linear Dynamical Systems
• Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon
• On the Analysis of Model-free Methods for the Linear Quadratic Regulator