Neural Auto-Curricula

Abstract

When solving two-player zero-sum games, multi-agent reinforcement learning (MARL) algorithms often create populations of agents where, at each iteration, a new agent is discovered as the best response to a mixture over the opponent population. Within such a process, the update rules of "who to compete with" (i.e., the opponent mixture) and "how to beat them" (i.e., finding best responses) are underpinned by manually developed game theoretical principles such as fictitious play and Double Oracle. In this paper, we introduce a novel framework -- Neural Auto-Curricula (NAC) -- that leverages meta-gradient descent to automate the discovery of the learning update rule without explicit human design. Specifically, we parameterise the opponent selection module by neural networks and the best-response module by optimisation subroutines, and update their parameters solely via interaction with the game engine, where both players aim to minimise their exploitability. Surprisingly, even without human design, the discovered MARL algorithms achieve competitive or even better performance with the state-of-the-art population-based game solvers (e.g., PSRO) on Games of Skill, differentiable Lotto, non-transitive Mixture Games, Iterated Matching Pennies, and Kuhn Poker. Additionally, we show that NAC is able to generalise from small games to large games, for example training on Kuhn Poker and outperforming PSRO on Leduc Poker. Our work inspires a promising future direction to discover general MARL algorithms solely from data.

arXiv commentcorresponding to <[email protected]>

References in corpus (26)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Learning to reinforcement learn
• Emergent Tool Use From Multi-Agent Autocurricula
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Game-Theoretic Multiagent Reinforcement Learning
• Evolved Policy Gradients
• OpenSpiel: A Framework for Reinforcement Learning in Games
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Discovering Reinforcement Learning Algorithms
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Open-Ended Learning Leads to Generally Capable Agents
• ES-MAML: Simple Hessian-Free Meta Learning
• Structured Evolution with Compact Architectures for Scalable Policy Optimization
• Discovery of Useful Questions as Auxiliary Tasks
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• A Self-Tuning Actor-Critic Algorithm
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Real World Games Look Like Spinning Tops
• Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Online Double Oracle