Mean Field Multi-Agent Reinforcement Learning

Abstract

Existing multi-agent reinforcement learning methods are limited typically to a small number of agents. When the agent number increases largely, the learning becomes intractable due to the curse of the dimensionality and the exponential growth of agent interactions. In this paper, we present \emph{Mean Field Reinforcement Learning} where the interactions within the population of agents are approximated by those between a single agent and the average effect from the overall population or neighboring agents; the interplay between the two entities is mutually reinforced: the learning of the individual agent's optimal policy depends on the dynamics of the population, while the dynamics of the population change according to the collective patterns of the individual policies. We develop practical mean field Q-learning and mean field Actor-Critic algorithms and analyze the convergence of the solution to Nash equilibrium. Experiments on Gaussian squeeze, Ising model, and battle games justify the learning effectiveness of our mean field approaches. In addition, we report the first result to solve the Ising model via model-free reinforcement learning methods.

ICML 2018 (Full paper + Long talk)

References in corpus (8)

• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Counterfactual Multi-Agent Policy Gradients
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Ising model versus normal form game
• MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence
• A Polynomial-time Nash Equilibrium Algorithm for Repeated Stochastic Games
• Learning Deep Mean Field Games for Modeling Large Population Behavior

Cited by in corpus (71)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning
• Game-Theoretic Multiagent Reinforcement Learning
• Survey on Machine Learning for Traffic-Driven Service Provisioning in Optical Networks
• Reward Design for Driver Repositioning Using Multi-Agent Reinforcement Learning
• Multi-Agent Reinforcement Learning for Markov Routing Games: A New Modeling Paradigm For Dynamic Traffic Assignment
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
• Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents
• Deep Coordination Graphs
• A DRL-based Multiagent Cooperative Control Framework for CAV Networks: a Graphic Convolution Q Network
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods
• Human-Centric Resource Allocation for the Metaverse With Multiaccess Edge Computing
• Multi-Agent Distributed Reinforcement Learning for Making Decentralized Offloading Decisions
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers
• A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising
• Settling the Variance of Multi-Agent Policy Gradients
• Cellular Network Capacity and Coverage Enhancement with MDT Data and Deep Reinforcement Learning
• Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games
• Stabilizing Traffic via Autonomous Vehicles: A Continuum Mean Field Game Approach
• Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing
• Distributed Mean-Field Density Estimation for Large-Scale Systems
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey
• Agent-Based Campus Novel Coronavirus Infection and Control Simulation
• Action Semantics Network: Considering the Effects of Actions in Multiagent Systems
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Learning in Nonzero-Sum Stochastic Games with Potentials
• Efficient Collaborative Multi-Agent Deep Reinforcement Learning for Large-Scale Fleet Management
• Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint
• A Multi-Agent Reinforcement Learning Method for Impression Allocation in Online Display Advertising
• Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator
• Learning Mean-Field Games
• Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning
• Deep Reinforcement Learning for Demand Driven Services in Logistics and Transportation Systems: A Survey
• Graph Exploration for Effective Multi-agent Q-Learning
• Combinational Q-Learning for Dou Di Zhu
• ${\alpha }^{\alpha }$-Rank: Practically Scaling $\alpha$-Rank through Stochastic Optimisation
• Multi-Agent Reinforcement Learning for Long-Term Network Resource Allocation through Auction: a V2X Application
• Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems
• Optimizing Large-Scale Fleet Management on a Road Network using Multi-Agent Deep Reinforcement Learning with Graph Neural Network
• Reinforcement Learning reveals fundamental limits on the mixing of active particles
• Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach
• Correlation-aware Cooperative Multigroup Broadcast 360° Video Delivery Network: A Hierarchical Deep Reinforcement Learning Approach
• Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning
• Causal Mean Field Multi-Agent Reinforcement Learning
• Bi-level Actor-Critic for Multi-agent Coordination
• Adaptive Online Distributed Optimal Control of Very-Large-Scale Robotic Systems
• Survey on Multi-Agent Q-Learning frameworks for resource management in wireless sensor network
• Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning
• Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation
• Reinforcement Learning for Mean Field Games with Strategic Complementarities
• Neural MMO v1.3: A Massively Multiagent Game Environment for Training and Evaluating Neural Networks
• Coding for Distributed Multi-Agent Reinforcement Learning
• Battlesnake Challenge: A Multi-agent Reinforcement Learning Playground with Human-in-the-loop
• Lyapunov-Based Reinforcement Learning for Decentralized Multi-Agent Control
• Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph
• Adversarial Attacks in Cooperative AI
• Entropy based Independent Learning in Anonymous Multi-Agent Settings
• Shaping Large Population Agent Behaviors Through Entropy-Regularized Mean-Field Games
• Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration
• Mean Field MARL Based Bandwidth Negotiation Method for Massive Devices Spectrum Sharing
• Infer Your Enemies and Know Yourself, Learning in Real-Time Bidding with Partially Observable Opponents
• Simulation-Based Inference for Global Health Decisions
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Mean Field Equilibrium in Multi-Armed Bandit Game with Continuous Reward
• Voting-Based Multi-Agent Reinforcement Learning for Intelligent IoT