Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

Abstract

We explore reinforcement learning methods for finding the optimal policy in the linear quadratic regulator (LQR) problem. In particular, we consider the convergence of policy gradient methods in the setting of known and unknown parameters. We are able to produce a global linear convergence guarantee for this approach in the setting of finite time horizon and stochastic state dynamics under weak assumptions. The convergence of a projected policy gradient method is also established in order to handle problems with constraints. We illustrate the performance of the algorithm with two examples. The first example is the optimal liquidation of a holding in an asset. We show results for the case where we assume a model for the underlying dynamics and where we apply the method to the data directly. The empirical evidence suggests that the policy gradient method can learn the global optimal solution for a larger class of stochastic systems containing the LQR framework and that it is more robust with respect to model mis-specification when compared to a model-based approach. The second example is an LQR system in a higher dimensional setting with synthetic data.

arXiv comment49 pages, 9 figures

References in corpus (11)

• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• A reinforcement learning extension to the Almgren-Chriss model for optimal trade execution
• LQR through the Lens of First Order Methods: Discrete-time Case
• Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
• Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods
• Thompson Sampling for Linear-Quadratic Control Problems
• On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
• Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games
• Entropy Regularization for Mean Field Games with Learning
• Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Control
• On the Analysis of Model-free Methods for the Linear Quadratic Regulator

Cited by in corpus (4)

• Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon
• Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
• Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with $\sqrt{T}$ Regret
• Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games