Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods

Abstract

We investigate reinforcement learning in the setting of Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Applications include, for example, the control of a large number of robots communicating through a central unit dispatching the optimal policy computed by maximizing an aggregate reward. An approximate solution is obtained by learning the optimal policy of a generic agent interacting with the statistical distribution of the states and actions of the other agents. We first provide a full analysis this discrete-time mean field control problem. We then rigorously prove the convergence of exact and model-free policy gradient methods in a mean-field linear-quadratic setting and establish bounds on the rates of convergence. We also provide graphical evidence of the convergence based on implementations of our algorithms.

References in corpus (3)

• Mean Field Multi-Agent Reinforcement Learning
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• Reinforcement Learning for Mean Field Game

Cited by in corpus (16)

• Game-Theoretic Multiagent Reinforcement Learning
• Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning
• Dynamic Programming Principles for Mean-Field Controls with Learning
• Mean Field Markov Decision Processes
• Entropy Regularization for Mean Field Games with Learning
• Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon
• Learning in Discounted-cost and Average-cost Mean-field Games
• Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games
• On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)
• Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
• Exploration noise for learning linear-quadratic mean field games
• Adaptive Online Distributed Optimal Control of Very-Large-Scale Robotic Systems
• Model Free Reinforcement Learning Algorithm for Stationary Mean field Equilibrium for Multiple Types of Agents
• Reinforcement Learning for Mean Field Games with Strategic Complementarities
• Policy Optimization for Linear-Quadratic Zero-Sum Mean-Field Type Games
• Adversarial Attacks in Cooperative AI