Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization
arXiv:2103.04564
Abstract
We propose a simple, general and effective technique, Reward Randomization for discovering diverse strategic policies in complex multi-agent games. Combining reward randomization and policy gradient, we derive a new algorithm, Reward-Randomized Policy Gradient (RPG). RPG is able to discover multiple distinctive human-interpretable strategies in challenging temporal trust dilemmas, including grid-world games and a real-world game Agar.io, where multiple equilibria exist but standard multi-agent policy gradient algorithms always converge to a fixed one with a sub-optimal payoff for every player even using state-of-the-art exploration techniques. Furthermore, with the set of diverse strategies from RPG, we can (1) achieve higher payoffs by fine-tuning the best policy from the set; and (2) obtain an adaptive agent by using this set of strategies as its training opponents. The source code and example videos can be found in our website: https://sites.google.com/view/staghuntrpg.
Accepted paper on ICLR 2021. First two authors share equal contribution
References in corpus (12)
- Robots that can adapt like animals
- StarCraft II: A New Challenge for Reinforcement Learning
- Learning to Navigate in Complex Environments
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Building Generalizable Agents with a Realistic and Rich 3D Environment
- MAVEN: Multi-Agent Variational Exploration
- Provable Self-Play Algorithms for Competitive Reinforcement Learning
- Open-ended Learning in Symmetric Zero-sum Games
- "Other-Play" for Zero-Shot Coordination
- No-Regret Learning in Bayesian Games
- Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
- Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games