Robust Reinforcement Learning: A Case Study in Linear Quadratic Regulation

Abstract

This paper studies the robustness of reinforcement learning algorithms to errors in the learning process. Specifically, we revisit the benchmark problem of discrete-time linear quadratic regulation (LQR) and study the long-standing open question: Under what conditions is the policy iteration method robustly stable from a dynamical systems perspective? Using advanced stability results in control theory, it is shown that policy iteration for LQR is inherently robust to small errors in the learning process and enjoys small-disturbance input-to-state stability: whenever the error in each iteration is bounded and small, the solutions of the policy iteration algorithm are also bounded, and, moreover, enter and stay in a small neighbourhood of the optimal LQR solution. As an application, a novel off-policy optimistic least-squares policy iteration for the LQR problem is proposed, when the system dynamics are subjected to additive stochastic disturbances. The proposed new results in robust reinforcement learning are validated by a numerical example.

arXiv commentarXiv admin note: text overlap with arXiv:2005.09528

References in corpus (15)

• Continuous control with deep reinforcement learning
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Continuous Inverse Optimal Control with Locally Optimal Examples
• Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems
• LQR through the Lens of First Order Methods: Discrete-time Case
• Model-Free Linear Quadratic Control via Reduction to Expert Prediction
• Learning the Globally Optimal Distributed LQ Regulator
• Learning-based Control of Unknown Linear Systems with Thompson Sampling
• Characterizing the Exact Behaviors of Temporal Difference Learning Algorithms Using Markov Jump Linear System Theory
• On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
• Finite-Time Performance Bounds and Adaptive Learning Rate Selection for Two Time-Scale Reinforcement Learning
• Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon
• Combining Model-Based and Model-Free Methods for Nonlinear Control: A Provably Convergent Policy Gradient Approach
• Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator
• Robust Policy Iteration for Continuous-time Linear Quadratic Regulation