Safe Reinforcement Learning Using Robust MPC

Abstract

Reinforcement Learning (RL) has recently impressed the world with stunning results in various applications. While the potential of RL is now well-established, many critical aspects still need to be tackled, including safety and stability issues. These issues, while partially neglected by the RL community, are central to the control community which has been widely investigating them. Model Predictive Control (MPC) is one of the most successful control techniques because, among others, of its ability to provide such guarantees even for uncertain constrained systems. Since MPC is an optimization-based technique, optimality has also often been claimed. Unfortunately, the performance of MPC is highly dependent on the accuracy of the model used for predictions. In this paper, we propose to combine RL and MPC in order to exploit the advantages of both and, therefore, obtain a controller which is optimal and safe. We illustrate the results with a numerical example in simulations.

References in corpus (4)

• Safe Exploration in Continuous Action Spaces
• Data-driven Economic NMPC using Reinforcement Learning
• Differentiable MPC for End-to-end Planning and Control
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning

Cited by in corpus (41)

• Data-Driven Model Predictive Control with Stability and Robustness Guarantees
• Robust Learning-based Predictive Control for Discrete-time Nonlinear Systems with Unknown Dynamics and State Constraints
• A Review of Safe Reinforcement Learning Methods for Modern Power Systems
• A Reinforcement Learning-based Economic Model Predictive Control Framework for Autonomous Operation of Chemical Reactors
• On the design of terminal ingredients for data-driven MPC
• Koopman based data-driven predictive control
• Reinforcement Learning based on Scenario-tree MPC for ASVs
• The Implicit Rigid Tube Model Predictive Control
• Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems
• Near-Optimal Design of Safe Output Feedback Controllers from Noisy Data
• DiffTune-MPC: Closed-Loop Learning for Model Predictive Control
• A Survey of Reinforcement Learning for Optimization in Automation
• Neural Lyapunov Differentiable Predictive Control
• Optimal Management of the Peak Power Penalty for Smart Grids Using MPC-based Reinforcement Learning
• Multi-Agent Reinforcement Learning via Distributed MPC as a Function Approximator
• Lagrangian-based online safe reinforcement learning for state-constrained systems
• Learning safety in model-based Reinforcement Learning using MPC and Gaussian Processes
• Verification of Dissipativity and Evaluation of Storage Function in Economic Nonlinear MPC using Q-Learning
• Learning to Boost the Performance of Stable Nonlinear Systems
• Safety Critical Control for Nonlinear Systems with Complex Input Constraints
• Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part II - Deterministic Case
• Constrained Model-Free Reinforcement Learning for Process Optimization
• Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging
• Online Optimal Control with Affine Constraints
• Learning Model Predictive Control Parameters via Bayesian Optimization for Battery Fast Charging
• Imitation Learning of MPC with Neural Networks: Error Guarantees and Sparsification
• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guarantee
• A modular framework for stabilizing deep reinforcement learning control
• Actor-Critic Cooperative Compensation to Model Predictive Control for Off-Road Autonomous Vehicles Under Unknown Dynamics
• Approximate solution of stochastic infinite horizon optimal control problems for constrained linear uncertain systems
• Two-step reinforcement learning for model-free redesign of nonlinear optimal regulator
• MPC-based Reinforcement Learning for a Simplified Freight Mission of Autonomous Surface Vehicles
• Learning Quasi-LPV Models and Robust Control Invariant Sets with Reduced Conservativeness
• Exploiting Prior Knowledge in Preferential Learning of Individualized Autonomous Vehicle Driving Styles
• A Contingency Model Predictive Control Framework for Safe Learning
• Data Generation Method for Learning a Low-dimensional Safe Region in Safe Reinforcement Learning
• Assured RL: Reinforcement Learning with Almost Sure Constraints
• LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World
• A predictive modular approach to constraint satisfaction under uncertainty -- with application to glycosylation in continuous monoclonal antibody biosimilar production
• Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)