Learning in Nonzero-Sum Stochastic Games with Potentials

Abstract

Multi-agent reinforcement learning (MARL) has become effective in tackling discrete cooperative game scenarios. However, MARL has yet to penetrate settings beyond those modelled by team and zero-sum games, confining it to a small subset of multi-agent systems. In this paper, we introduce a new generation of MARL learners that can handle nonzero-sum payoff structures and continuous settings. In particular, we study the MARL problem in a class of games known as stochastic potential games (SPGs) with continuous state-action spaces. Unlike cooperative games, in which all agents share a common reward, SPGs are capable of modelling real-world scenarios where agents seek to fulfil their individual goals. We prove theoretically our learning method, SPot-AC, enables independent agents to learn Nash equilibrium strategies in polynomial time. We demonstrate our framework tackles previously unsolvable tasks such as Coordination Navigation and large selfish routing games and that it outperforms the state of the art MARL baselines such as MADDPG and COMIX in such scenarios.

arXiv commentICML 2021

References in corpus (16)

• Continuous control with deep reinforcement learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Counterfactual Multi-Agent Policy Gradients
• Mean Field Multi-Agent Reinforcement Learning
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• Flows and Decompositions of Games: Harmonic and Potential Games
• Game-Theoretic Multiagent Reinforcement Learning
• The Mechanics of n-Player Differentiable Games
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Global Convergence of Multi-Agent Policy Gradient in Markov Potential Games
• Learning Parametric Closed-Loop Policies for Markov Potential Games
• Dynamic Potential Games in Communications: Fundamentals and Applications
• ${\alpha }^{\alpha }$-Rank: Practically Scaling $\alpha$-Rank through Stochastic Optimisation

Cited by in corpus (1)

• Learning to Shape Rewards using a Game of Two Partners