Game-Theoretic Multiagent Reinforcement Learning

Abstract

Tremendous advances have been made in multiagent reinforcement learning (MARL). MARL corresponds to the learning problem in a multiagent system in which multiple agents learn simultaneously. It is an interdisciplinary field of study with a long history that includes game theory, machine learning, stochastic control, psychology, and optimization. Despite great successes in MARL, there is a lack of a self-contained overview of the literature that covers game-theoretic foundations of modern MARL methods and summarizes the recent advances. The majority of existing surveys are outdated and do not fully cover the recent developments since 2010. In this work, we provide a monograph on MARL that covers both the fundamentals and the latest developments on the research frontier. The goal of this monograph is to provide a self-contained assessment of the current state-of-the-art MARL techniques from a game-theoretic perspective. We expect this work to serve as a stepping stone for both new researchers who are about to enter this fast-growing field and experts in the field who want to obtain a panoramic view and identify new directions based on recent advances.

References in corpus (79)

• Deep Learning in Neural Networks: An Overview
• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Language Models are Few-Shot Learners
• Trust Region Policy Optimization
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
• Which Training Methods for GANs do actually Converge?
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• Distributed Optimal Power Flow for Smart Microgrids
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• StarCraft II: A New Challenge for Reinforcement Learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Counterfactual Multi-Agent Policy Gradients
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Latent Dynamics for Planning from Pixels
• Mean Field Multi-Agent Reinforcement Learning
• Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
• More is the Same; Phase Transitions and Mean Field Theories
• Complexity Results about Nash Equilibria
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Flows and Decompositions of Games: Harmonic and Potential Games
• On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
• A probabilistic weak formulation of mean field games and applications
• The Numerics of GANs
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• MAA*: A Heuristic Search Algorithm for Solving Decentralized POMDPs
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• The Mechanics of n-Player Differentiable Games
• Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
• Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods
• Mean field games via controlled martingale problems: Existence of Markovian equilibria
• Hybrid Block Successive Approximation for One-Sided Non-Convex Min-Max Problems: Algorithms and Applications
• A Theoretical Analysis of Deep Q-Learning
• Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile
• A unified view of entropy-regularized Markov decision processes
• Policy Iteration for Decentralized Control of Markov Decision Processes
• On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games
• Hierarchical POMDP Controller Optimization by Likelihood Maximization
• Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
• What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Open-ended Learning in Symmetric Zero-sum Games
• Finite mean field games: fictitious play and convergence to a first order continuous mean field game
• Online Convex Optimization in Adversarial Markov Decision Processes
• No-regret Dynamics and Fictitious Play
• Convergence of Learning Dynamics in Stackelberg Games
• Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning
• Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games with Local Control and Global State Information
• Re-evaluating Evaluation
• Wasserstein Robust Reinforcement Learning
• Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods
• Solving Discounted Stochastic Two-Player Games with Near-Optimal Time and Sample Complexity
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• Efficient Algorithms for Smooth Minimax Optimization
• Variational Regret Bounds for Reinforcement Learning
• Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Feature-Based Q-Learning for Two-Player Stochastic Games
• Rollout Sampling Policy Iteration for Decentralized POMDPs
• A Mean Field Game of Portfolio Trading and Its Consequences On Perceived Correlations
• Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games
• Dynamic Programming Principles for Mean-Field Controls with Learning
• Convergence of Monte Carlo Tree Search in Simultaneous Move Games
• An accelerated inexact proximal point method for solving nonconvex-concave min-max problems
• Neural Replicator Dynamics
• Learning in Discounted-cost and Average-cost Mean-field Games
• Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator
• A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
• Multi-Agent Determinantal Q-Learning
• Dynamic Potential Games in Communications: Fundamentals and Applications
• Convergence Analysis of Gradient-Based Learning with Non-Uniform Learning Rates in Non-Cooperative Multi-Agent Settings
• Stochastic Potential Games

Cited by in corpus (17)

• Social Interactions for Autonomous Driving: A Review and Perspectives
• Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
• Coordination for Connected and Automated Vehicles at Non-signalized Intersections: A Value Decomposition-based Multiagent Deep Reinforcement Learning Approach
• On the Complexity of Computing Markov Perfect Equilibrium in General-Sum Stochastic Games
• Emergence of Cooperation in Two-agent Repeated Games with Reinforcement Learning
• Settling the Variance of Multi-Agent Policy Gradients
• Multi-Agent Constrained Policy Optimisation
• Learning in Nonzero-Sum Stochastic Games with Potentials
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
• Graph Attention Multi-Agent Fleet Autonomy for Advanced Air Mobility
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
• A Game-Theoretic Approach for Improving Generalization Ability of TSP Solvers
• Online Markov Decision Processes with Non-oblivious Strategic Adversary
• DM${}^2$: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching
• Multi-agent Reinforcement Learning in OpenSpiel: A Reproduction Report
• Neural Auto-Curricula