Emergent Coordination Through Competition

Abstract

We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.

Cited by in corpus (32)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• dm_control: Software and Tasks for Continuous Control
• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Meta reinforcement learning as task inference
• Effective Diversity in Population Based Reinforcement Learning
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• A Generalized Training Approach for Multiagent Learning
• Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Emergent Road Rules In Multi-Agent Driving Environments
• Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
• Collective Intelligence for Deep Learning: A Survey of Recent Developments
• ${\alpha }^{\alpha }$-Rank: Practically Scaling $\alpha$-Rank through Stochastic Optimisation
• Coordination in Adversarial Sequential Team Games via Multi-Agent Deep Reinforcement Learning
• SAT-MARL: Specification Aware Training in Multi-Agent Reinforcement Learning
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• Multi-Agent Coordination in Adversarial Environments through Signal Mediated Strategies
• Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space
• Efficient Competitive Self-Play Policy Optimization
• Modeling Sensorimotor Coordination as Multi-Agent Reinforcement Learning with Differentiable Communication
• Universal Policies to Learn Them All