Multi-Agent Reinforcement Learning via Distributed MPC as a Function Approximator

Abstract

This paper presents a novel approach to multi-agent reinforcement learning (RL) for linear systems with convex polytopic constraints. Existing work on RL has demonstrated the use of model predictive control (MPC) as a function approximator for the policy and value functions. The current paper is the first work to extend this idea to the multi-agent setting. We propose the use of a distributed MPC scheme as a function approximator, with a structure allowing for distributed learning and deployment. We then show that Q-learning updates can be performed distributively without introducing nonstationarity, by reconstructing a centralized learning update. The effectiveness of the approach is demonstrated on two numerical examples.

12 pages, 8 figures, accepted for publication in Automatica, code can be found at https://github.com/SamuelMallick/dmpcrl-concept/tree/paper-2023

References in corpus (7)

• A Brief Survey of Deep Reinforcement Learning
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Safe Reinforcement Learning Using Robust MPC
• The Scenario Approach for Stochastic Model Predictive Control with Bounds on Closed-Loop Constraint Violations
• Data-driven Economic NMPC using Reinforcement Learning
• A Proof of Convergence For the Alternating Direction Method of Multipliers Applied to Polyhedral-Constrained Functions
• Learning safety in model-based Reinforcement Learning using MPC and Gaussian Processes