Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination

Abstract

Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward, as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Furthermore, relying solely on the agent-specific reward is sub-optimal because it usually does not capture the team coordination objective. A common approach is to use reward shaping to construct a proxy reward by combining the individual rewards. However, this requires manual tuning for each environment. We introduce Multiagent Evolutionary Reinforcement Learning (MERL), a split-level training platform that handles the two objectives separately through two optimization processes. An evolutionary algorithm maximizes the sparse team-based objective through neuroevolution on a population of teams. Concurrently, a gradient-based optimizer trains policies to only maximize the dense agent-specific rewards. The gradient-based policies are periodically added to the evolutionary population as a way of information transfer between the two optimization processes. This enables the evolutionary algorithm to use skills learned via the agent-specific rewards toward optimizing the global objective. Results demonstrate that MERL significantly outperforms state-of-the-art methods, such as MADDPG, on a number of difficult coordination benchmarks.

arXiv commentProceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 108, 2020

References in corpus (13)

• Continuous control with deep reinforcement learning
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Emergent Coordination Through Competition
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• A multi-agent reinforcement learning model of common-pool resource appropriation
• Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie

Cited by in corpus (5)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Evolutionary Reinforcement Learning: A Survey
• A Generalized Training Approach for Multiagent Learning
• Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning
• Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity