On Architectures for Combining Reinforcement Learning and Model Predictive Control with Runtime Improvements

Abstract

Model Predictive Control (MPC) faces computational demands and performance degradation from model inaccuracies. We propose two architectures combining Neural Network-approximated MPC (NNMPC) with Reinforcement Learning (RL). The first, Warm Start RL, initializes the RL actor with pre-trained NNMPC weights. The second, RLMPC, uses RL to generate corrective residuals for NNMPC outputs. We introduce a downsampling method reducing NNMPC input dimensions while maintaining performance. Evaluated on a rotary inverted pendulum, both architectures demonstrate runtime reductions exceeding 99% compared to traditional MPC while improving tracking performance under model uncertainties, with RL+MPC achieving 11-40% cost reduction depending on reference amplitude.

Accepted at the 2025 IFAC Conference on Modeling, Estimation, and Control of Systems (MECC 2025), Pittsburgh, USA