Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

Abstract

Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation. Enhanced POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved through other means.

arXiv comment23 pages, 14 figures

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• How transferable are features in deep neural networks?
• Dota 2 with Large Scale Deep Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robots that can adapt like animals
• Emergence of Locomotion Behaviours in Rich Environments
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Illuminating search spaces by mapping elites
• Learning to reinforcement learn
• Go-Explore: a New Approach for Hard-Exploration Problems
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Emergent Complexity via Multi-Agent Competition
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods

Cited by in corpus (20)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Navigating the Landscape of Multiplayer Games
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
• Few-shot Quality-Diversity Optimization
• A Re-classification of Information Seeking Tasks and Their Computational Solutions
• Active Divergence with Generative Deep Learning -- A Survey and Taxonomy
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Adversarial Environment Generation for Learning to Navigate the Web
• Self-Organizing Intelligent Matter: A blueprint for an AI generating algorithm
• Simple Genetic Operators are Universal Approximators of Probability Distributions (and other Advantages of Expressive Encodings)
• A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Prioritized Level Replay
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Open-ended search for environments and adapted agents using MAP-Elites
• Evolutionary Self-Replication as a Mechanism for Producing Artificial Intelligence
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks