Learning to Play No-Press Diplomacy with Best Response Policy Iteration
arXiv:2006.04635
Abstract
Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually simple and principled application of RL methods. However real-world settings are many-agent, and agent interactions are complex mixtures of common-interest and competitive aspects. We consider Diplomacy, a 7-player board game designed to accentuate dilemmas resulting from many-agent interactions. It also features a large combinatorial action space and simultaneous moves, which are challenging for RL algorithms. We propose a simple yet effective approximate best response operator, designed to handle large combinatorial action spaces and simultaneous moves. We also introduce a family of policy iteration methods that approximate fictitious play. With these methods, we successfully apply RL to Diplomacy: we show that our agents convincingly outperform the previous state-of-the-art, and game theoretic equilibrium analysis shows that the new process yields consistent improvements.
References in corpus (26)
- Playing Atari with Deep Reinforcement Learning
- DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Emergent Tool Use From Multi-Agent Autocurricula
- The Hanabi Challenge: A New Frontier for AI Research
- Cooperating with Machines
- Thinking Fast and Slow with Deep Learning and Tree Search
- Emergent Complexity via Multi-Agent Competition
- Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
- Maintaining cooperation in complex social dilemmas using deep reinforcement learning
- OpenSpiel: A Framework for Reinforcement Learning in Games
- On the Utility of Learning about Humans for Human-AI Coordination
- Emergent Coordination Through Competition
- Open-ended Learning in Symmetric Zero-sum Games
- The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
- Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
- Re-evaluating Evaluation
- A Counter-Example to Karlin's Strong Conjecture for Fictitious Play
- Learning to Correlate in Multi-Player General-Sum Sequential Games
- Finding Friend and Foe in Multi-Agent Games
- From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization
- Equilibrium Approximation Quality of Current No-Limit Poker Bots
- No Press Diplomacy: Modeling Multi-Agent Gameplay
- Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
- Regret Minimization in Non-Zero-Sum Games with Applications to Building Champion Multiplayer Computer Poker Agents