Emergent Tool Use From Multi-Agent Autocurricula

Abstract

Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.

References in corpus (24)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Emergence of Locomotion Behaviours in Rich Environments
• VIME: Variational Information Maximizing Exploration
• Counterfactual Multi-Agent Policy Gradients
• Competitive Coevolution through Evolutionary Complexification
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Learning Dexterous In-Hand Manipulation
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Relational Deep Reinforcement Learning
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Emergent Coordination Through Competition
• A multi-agent reinforcement learning model of common-pool resource appropriation
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Requirements for Open-Ended Evolution in Natural and Artificial Systems

Cited by in corpus (94)

• On the Opportunities and Risks of Foundation Models
• Unity: A General Platform for Intelligent Agents
• Collision Avoidance in Pedestrian-Rich Environments with Deep Reinforcement Learning
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Game-Theoretic Multiagent Reinforcement Learning
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
• Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number
• Automatic Curriculum Learning through Value Disagreement
• On the Critical Role of Conventions in Adaptive Human-AI Collaboration
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Spatial Intention Maps for Multi-Agent Mobile Manipulation
• One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control
• A Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation Assurance
• Creative Problem Solving in Artificially Intelligent Agents: A Survey and Framework
• Fair collaborative vehicle routing: A deep multi-agent reinforcement learning approach
• Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning
• Potential Impacts of Smart Homes on Human Behavior: A Reinforcement Learning Approach
• SquirRL: Automating Attack Analysis on Blockchain Incentive Mechanisms with Deep Reinforcement Learning
• Multi-Agent Collaboration via Reward Attribution Decomposition
• Learning a Decentralized Multi-arm Motion Planner
• Reinforcement Learning for Photonic Component Design
• Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• Open Problems in Cooperative AI
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning
• Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
• Explaining a Deep Reinforcement Learning Docking Agent Using Linear Model Trees with User Adapted Visualization
• Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Tasks
• Obtaining Robust Control and Navigation Policies for Multi-Robot Navigation via Deep Reinforcement Learning
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Learning Generalizable Visual Representations via Interactive Gameplay
• Building Human-like Communicative Intelligence: A Grounded Perspective
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
• Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning
• Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• A Mathematical Theory of Attention
• Learning to Track Dynamic Targets in Partially Known Environments
• Joint Policy Search for Multi-agent Collaboration with Imperfect Information
• Emergent Road Rules In Multi-Agent Driving Environments
• When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?
• Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning
• Collective Intelligence for Deep Learning: A Survey of Recent Developments
• A Sharp Analysis of Model-based Reinforcement Learning with Self-Play
• On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning
• Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems
• Few Shot System Identification for Reinforcement Learning
• Reinforcement Learning with Success Induced Task Prioritization
• Learning to Deceive in Multi-Agent Hidden Role Games
• Low-Bandwidth Communication Emerges Naturally in Multi-Agent Learning Systems
• Disentangling causal effects for hierarchical reinforcement learning
• Hierarchically Decoupled Imitation for Morphological Transfer
• Discovering Fatigued Movements for Virtual Character Animation
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Reinforcement Learning Based Self-play and State Stacking Techniques for Noisy Air Combat Environment
• Competing AI: How does competition feedback affect machine learning?
• Guided Exploration with Proximal Policy Optimization using a Single Demonstration
• Learning to solve arithmetic problems with a virtual abacus
• On the Regularity of Attention
• Co-Training an Observer and an Evading Target
• Learning cooperative behaviours in adversarial multi-agent systems
• Neural MMO v1.3: A Massively Multiagent Game Environment for Training and Evaluating Neural Networks
• Reinforcement Learning Agents for Ubisoft's Roller Champions
• A Generative Model based Adversarial Security of Deep Learning and Linear Classifier Models
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historical Context and Future Challenges
• Deep Policy Networks for NPC Behaviors that Adapt to Changing Design Parameters in Roguelike Games
• Minimizing Communication while Maximizing Performance in Multi-Agent Reinforcement Learning
• Learning Provably Robust Motion Planners Using Funnel Libraries
• Visual Perspective Taking for Opponent Behavior Modeling
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Learning to Actively Reduce Memory Requirements for Robot Control Tasks
• Learning to Play Foosball: System and Baselines
• Neural Auto-Curricula
• Individual and Collective Autonomous Development
• Behaviour-conditioned policies for cooperative reinforcement learning tasks
• Meta-control of social learning strategies
• Explore and Control with Adversarial Surprise
• Towards sample-efficient episodic control with DAC-ML
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning
• Complex Momentum for Optimization in Games