Stable Opponent Shaping in Differentiable Games

Abstract

A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -- from GANs and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player influence on others' updates. Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner's Dilemma. Although experimentally successful, we show that LOLA agents can exhibit 'arrogant' behaviour directly at odds with convergence. In fact, remarkably few algorithms have theoretical guarantees applying across all (n-player, non-convex) games. In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead. We prove that LookAhead converges locally to equilibria and avoids strict saddles in all differentiable games. SOS inherits these essential guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally.

arXiv comment20 pages, 7 figures

References in corpus (6)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• The Numerics of GANs
• The Mechanics of n-Player Differentiable Games
• First-order Methods Almost Always Avoid Saddle Points
• Global Convergence to the Equilibrium of GANs using Variational Inequalities
• DiCE: The Infinitely Differentiable Monte-Carlo Estimator

Cited by in corpus (6)

• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Open Problems in Cooperative AI
• Cooperation and Reputation Dynamics with Reinforcement Learning
• Learning in two-player games between transparent opponents
• Behaviour-conditioned policies for cooperative reinforcement learning tasks