Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach

Abstract

Multi-agent reinforcement learning (MARL) becomes more challenging in the presence of more agents, as the capacity of the joint state and action spaces grows exponentially in the number of agents. To address such a challenge of scale, we identify a class of cooperative MARL problems with permutation invariance, and formulate it as a mean-field Markov decision processes (MDP). To exploit the permutation invariance therein, we propose the mean-field proximal policy optimization (MF-PPO) algorithm, at the core of which is a permutation-invariant actor-critic neural architecture. We prove that MF-PPO attains the globally optimal policy at a sublinear rate of convergence. Moreover, its sample complexity is independent of the number of agents. We validate the theoretical advantages of MF-PPO with numerical experiments in the multi-agent particle environment (MPE). In particular, we show that the inductive bias introduced by the permutation-invariant neural architecture enables MF-PPO to outperform existing competitors with a smaller number of model parameters, which is the key to its generalization performance.

References in corpus (13)

• Semi-Supervised Classification with Graph Convolutional Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Kernel Mean Embedding of Distributions: A Review and Beyond
• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Mean Field Multi-Agent Reinforcement Learning
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
• Deep Reinforcement Learning for Event-Driven Multi-Agent Decision Processes
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning
• Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning