A Game-Theoretic Approach to Multi-Agent Trust Region Optimization

Abstract

Trust region methods are widely applied in single-agent reinforcement learning problems due to their monotonic performance-improvement guarantee at every iteration. Nonetheless, when applied in multi-agent settings, the guarantee of trust region methods no longer holds because an agent's payoff is also affected by other agents' adaptive behaviors. To tackle this problem, we conduct a game-theoretical analysis in the policy space, and propose a multi-agent trust region learning method (MATRL), which enables trust region optimization for multi-agent learning. Specifically, MATRL finds a stable improvement direction that is guided by the solution concept of Nash equilibrium at the meta-game level. We derive the monotonic improvement guarantee in multi-agent settings and empirically show the local convergence of MATRL to stable fixed points in the two-player rotational differential game. To test our method, we evaluate MATRL in both discrete and continuous multiplayer general-sum games including checker and switch grid worlds, multi-agent MuJoCo, and Atari games. Results suggest that MATRL significantly outperforms strong multi-agent reinforcement learning baselines.

A Multi-Agent Trust Region Learning (MATRL) algorithm that augments the single-agent trust region policy optimization with a weak stable fixed point approximated by the policy-space meta-game

References in corpus (7)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• On Finding Local Nash Equilibria (and Only Local Nash Equilibria) in Zero-Sum Games
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Existence of Multiagent Equilibria with Limited Agents
• Independent Policy Gradient Methods for Competitive Reinforcement Learning