A Tour of Reinforcement Learning: The View from Continuous Control

Abstract

This manuscript surveys reinforcement learning from the perspective of optimization and control with a focus on continuous control applications. It surveys the general formulation, terminology, and typical experimental implementations of reinforcement learning and reviews competing solution paradigms. In order to compare the relative merits of various techniques, this survey presents a case study of the Linear Quadratic Regulator (LQR) with unknown dynamics, perhaps the simplest and best-studied problem in optimal control. The manuscript describes how merging techniques from learning theory and control can provide non-asymptotic characterizations of LQR performance and shows that these characterizations tend to match experimental behavior. In turn, when revisiting more complex applications, many of the observed phenomena in LQR persist. In particular, theory and experiment demonstrate the role and importance of models and the cost of generality in reinforcement learning algorithms. This survey concludes with a discussion of some of the challenges in designing learning systems that safely and reliably interact with complex and uncertain environments and how tools from reinforcement learning and control might be combined to approach these challenges.

minor revision with a few clarifying passages and corrected typos

References in corpus (9)

• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Simple random search provides a competitive approach to reinforcement learning
• Near-Optimal Algorithms for Online Matrix Prediction
• Learning-based Control of Unknown Linear Systems with Thompson Sampling
• Thompson Sampling for Linear-Quadratic Control Problems
• Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator
• Bayesian Optimal Control of Smoothly Parameterized Systems: The Lazy Posterior Sampling Algorithm

Cited by in corpus (12)

• On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
• Exploration versus exploitation in reinforcement learning: a stochastic control approach
• On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator
• The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint
• Analyzing the Variance of Policy Gradient Estimators for the Linear-Quadratic Regulator
• Fast Online Reinforcement Learning Control using State-Space Dimensionality Reduction
• Personalized Cancer Chemotherapy Schedule: a numerical comparison of performance and robustness in model-based and model-free scheduling methodologies
• Provably Correct Learning Algorithms in the Presence of Time-Varying Features Using a Variational Perspective
• Aggressive Local Search for Constrained Optimal Control Problems with Many Local Minima
• Generalized maximum entropy estimation
• Online Algorithms and Policies Using Adaptive and Machine Learning Approaches
• Model-Free Optimal Control of Linear Multi-Agent Systems via Decomposition and Hierarchical Approximation