A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

Abstract

To achieve general intelligence, agents must learn how to interact with others in a shared environment: this is the challenge of multiagent reinforcement learning (MARL). The simplest form is independent reinforcement learning (InRL), where each agent treats its experience as part of its (non-stationary) environment. In this paper, we first observe that policies learned using InRL can overfit to the other agents' policies during training, failing to sufficiently generalize during execution. We introduce a new metric, joint-policy correlation, to quantify this effect. We describe an algorithm for general MARL, based on approximate best responses to mixtures of policies generated using deep reinforcement learning, and empirical game-theoretic analysis to compute meta-strategies for policy selection. The algorithm generalizes previous ones such as InRL, iterated best response, double oracle, and fictitious play. Then, we present a scalable implementation which reduces the memory requirement using decoupled meta-solvers. Finally, we demonstrate the generality of the resulting policies in two partially observable settings: gridworld coordination games and poker.

arXiv commentCamera-ready copy of NIPS 2017 paper, including appendix

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Learning Multiagent Communication with Backpropagation
• Emergent Complexity via Multi-Agent Competition
• Regret Minimization in Non-Zero-Sum Games with Applications to Building Champion Multiplayer Computer Poker Agents
• Using Reinforcement Learning to Validate Empirical Game-Theoretic Analysis: A Continuous Double Auction Study

Cited by in corpus (16)

• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning
• A social path to human-like artificial intelligence
• Navigating the Landscape of Multiplayer Games
• Quantifying the effects of environment and population diversity in multi-agent reinforcement learning
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• The Advantage Regret-Matching Actor-Critic
• StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning
• Non-cooperative Multi-agent Systems with Exploring Agents
• Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solvers
• Learning in two-player games between transparent opponents
• Search in Imperfect Information Games
• Paradigms of Computational Agency