Human-Level Performance in No-Press Diplomacy via Equilibrium Search

Abstract

Prior AI breakthroughs in complex games have focused on either the purely adversarial or purely cooperative settings. In contrast, Diplomacy is a game of shifting alliances that involves both cooperation and competition. For this reason, Diplomacy has proven to be a formidable research challenge. In this paper we describe an agent for the no-press variant of Diplomacy that combines supervised learning on human data with one-step lookahead search via regret minimization. Regret minimization techniques have been behind previous AI successes in adversarial games, most notably poker, but have not previously been shown to be successful in large-scale games involving cooperation. We show that our agent greatly exceeds the performance of past no-press Diplomacy bots, is unexploitable by expert humans, and ranks in the top 2% of human players when playing anonymous games on a popular Diplomacy website.

References in corpus (10)

• Dota 2 with Large Scale Deep Reinforcement Learning
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Fast Convergence of Regularized Learning in Games
• Combining Deep Reinforcement Learning and Search for Imperfect-Information Games
• Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• No Press Diplomacy: Modeling Multi-Agent Gameplay

Cited by in corpus (3)

• Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
• Safe Subgame Resolving for Extensive Form Correlated Equilibrium
• No-Press Diplomacy from Scratch