Open-Ended Learning Leads to Generally Capable Agents

Abstract

In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising to many held-out points in the universe of tasks. Examples of this zero-shot generalisation include good performance on Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks we characterise the behaviour of our agent, and find interesting emergent heuristic behaviours such as trial-and-error experimentation, simple tool use, option switching, and cooperation. Finally, we demonstrate that the general capabilities of this agent could unlock larger scale transfer of behaviour through cheap finetuning.

References in corpus (12)

• Dota 2 with Large Scale Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
• Understanding intermediate layers using linear classifier probes
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Kickstarting Deep Reinforcement Learning
• CPPN2GAN: Combining Compositional Pattern Producing Networks and GANs for Large-scale Pattern Generation
• Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• Contextual Games: Multi-Agent Learning with Side Information

Cited by in corpus (15)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• A social path to human-like artificial intelligence
• Sharing to learn and learning to share; Fitting together Meta-Learning, Multi-Task Learning, and Transfer Learning: A meta review
• Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory
• Transformers predicting the future. Applying attention in next-frame and time series forecasting
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• What Would Jiminy Cricket Do? Towards Agents That Behave Morally
• A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents
• Model based Multi-agent Reinforcement Learning with Tensor Decompositions
• Open-ended search for environments and adapted agents using MAP-Elites
• Meta-Learning Strategies through Value Maximization in Neural Networks
• Reinforcement Learning in Factored Action Spaces using Tensor Decompositions
• Neural Auto-Curricula
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms