Quantifying the effects of environment and population diversity in multi-agent reinforcement learning

Abstract

Generalization is a major challenge for multi-agent reinforcement learning. How well does an agent perform when placed in novel environments and in interactions with new co-players? In this paper, we investigate and quantify the relationship between generalization and diversity in the multi-agent domain. Across the range of multi-agent environments considered here, procedurally generating training levels significantly improves agent performance on held-out levels. However, agent performance on the specific levels used in training sometimes declines as a result. To better understand the effects of co-player variation, our experiments introduce a new environment-agnostic measure of behavioral diversity. Results demonstrate that population size and intrinsic motivation are both effective methods of generating greater population diversity. In turn, training with a diverse set of co-players strengthens agent performance in some (but not all) cases.

arXiv commentAccepted at Autonomous Agents and Multi-Agent Systems

References in corpus (9)

• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Reinforcement Learning with Deep Energy-Based Policies
• Quantifying Generalization in Reinforcement Learning
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
• Diversity is All You Need: Learning Skills without a Reward Function
• On the Utility of Learning about Humans for Human-AI Coordination
• Inequity aversion improves cooperation in intertemporal social dilemmas
• A multi-agent reinforcement learning model of common-pool resource appropriation

Cited by in corpus (2)

• Warmth and competence in human-agent cooperation
• A social path to human-like artificial intelligence