Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

Abstract

In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially.

arXiv commentThe project page is https://sites.google.com/view/epciclr2020 .The source code is released at https://github.com/qian18long/epciclr2020

References in corpus (16)

• Playing Atari with Deep Reinforcement Learning
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Emergent Tool Use From Multi-Agent Autocurricula
• Mean Field Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Relational Deep Reinforcement Learning
• Evolved Policy Gradients
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Emergent Coordination Through Competition
• Evolution-Guided Policy Gradient in Reinforcement Learning
• Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents
• Deep Multi-Agent Reinforcement Learning with Relevance Graphs

Cited by in corpus (6)

• Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
• Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
• On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning
• DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
• Learning Efficient Multi-Agent Cooperative Visual Exploration
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning