Data-driven Economic NMPC using Reinforcement Learning

Abstract

Reinforcement Learning (RL) is a powerful tool to perform data-driven optimal control without relying on a model of the system. However, RL struggles to provide hard guarantees on the behavior of the resulting control scheme. In contrast, Nonlinear Model Predictive Control (NMPC) and Economic NMPC (ENMPC) are standard tools for the closed-loop optimal control of complex systems with constraints and limitations, and benefit from a rich theory to assess their closed-loop behavior. Unfortunately, the performance of (E)NMPC hinges on the quality of the model underlying the control scheme. In this paper, we show that an (E)NMPC scheme can be tuned to deliver the optimal policy of the real system even when using a wrong model. This result also holds for real systems having stochastic dynamics. This entails that ENMPC can be used as a new type of function approximator within RL. Furthermore, we investigate our results in the context of ENMPC and formally connect them to the concept of dissipativity, which is central for the ENMPC stability. Finally, we detail how these results can be used to deploy classic RL tools for tuning (E)NMPC schemes. We apply these tools on both a classical linear MPC setting and a standard nonlinear example from the ENMPC literature.

References in corpus (1)

• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning

Cited by in corpus (28)

• Safe Reinforcement Learning Using Robust MPC
• Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control
• Safe Exploration in Model-based Reinforcement Learning using Control Barrier Functions
• A Reinforcement Learning-based Economic Model Predictive Control Framework for Autonomous Operation of Chemical Reactors
• Deep Model Predictive Variable Impedance Control
• Reinforcement Learning-based Model Predictive Control for Greenhouse Climate Control
• Reinforcement Learning based on Scenario-tree MPC for ASVs
• Constrained Controller and Observer Design by Inverse Optimality
• Multi-Agent Reinforcement Learning via Distributed MPC as a Function Approximator
• Optimal Management of the Peak Power Penalty for Smart Grids Using MPC-based Reinforcement Learning
• Learning safety in model-based Reinforcement Learning using MPC and Gaussian Processes
• End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control
• Verification of Dissipativity and Evaluation of Storage Function in Economic Nonlinear MPC using Q-Learning
• Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part II - Deterministic Case
• Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot
• Tutoring Reinforcement Learning via Feedback Control
• An Inverse Optimal Control Approach for Trajectory Prediction of Autonomous Race Cars
• Model predictive control-based value estimation for efficient reinforcement learning
• A view on learning robust goal-conditioned value functions: Interplay between RL and MPC
• Multi-agent Battery Storage Management using MPC-based Reinforcement Learning
• Oracle-based economic predictive control
• Computationally efficient Gauss-Newton reinforcement learning for model predictive control
• Reinforced Model Predictive Control via Trust-Region Quasi-Newton Policy Optimization
• Error-free approximation of explicit linear MPC through lattice piecewise affine expression
• MPC-based Reinforcement Learning for a Simplified Freight Mission of Autonomous Surface Vehicles
• Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)
• A predictive modular approach to constraint satisfaction under uncertainty -- with application to glycosylation in continuous monoclonal antibody biosimilar production
• Proximal Policy Optimization for Tracking Control Exploiting Future Reference Information