Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Abstract

While the history of machine learning so far largely encompasses a series of problems posed by researchers and algorithms that learn their solutions, an important question is whether the problems themselves can be generated by the algorithm at the same time as they are being solved. Such a process would in effect build its own diverse and expanding curricula, and the solutions to problems at various stages would become stepping stones towards solving even more challenging problems later in the process. The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper does just that: it pairs the generation of environmental challenges and the optimization of agents to solve those challenges. It simultaneously explores many different paths through the space of possible problems and solutions and, critically, allows these stepping-stone solutions to transfer between problems if better, catalyzing innovation. The term open-ended signifies the intriguing potential for algorithms like POET to continue to create novel and increasingly complex capabilities without bound. Our results show that POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved by direct optimization alone, or even through a direct-path curriculum-building control algorithm introduced to highlight the critical role of open-endedness in solving ambitious challenges. The ability to transfer solutions from one environment to another proves essential to unlocking the full potential of the system as a whole, demonstrating the unpredictable nature of fortuitous stepping stones. We hope that POET will inspire a new push towards open-ended discovery across many domains, where algorithms like POET can blaze a trail through their interesting possible manifestations and solutions.

28 pages, 9 figures

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Trust Region Policy Optimization
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robots that can adapt like animals
• Emergence of Locomotion Behaviours in Rich Environments
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Illuminating search spaces by mapping elites
• Episodic Curiosity through Reachability
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Emergent Complexity via Multi-Agent Competition
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Unsupervised Meta-Learning for Reinforcement Learning
• On the Relationship Between the OpenAI Evolution Strategy and Stochastic Gradient Descent
• Multi-task Deep Reinforcement Learning with PopArt

Cited by in corpus (53)

• Unity: A General Platform for Intelligent Agents
• Emergent Tool Use From Multi-Agent Autocurricula
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Societal Implications and Responsible Governance
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data
• Open-ended Learning in Symmetric Zero-sum Games
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• Safe Reinforcement Learning via Curriculum Induction
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Automated curricula through setter-solver interactions
• Creative Problem Solving in Artificially Intelligent Agents: A Survey and Framework
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
• Transparent Serverless execution of Python multiprocessing applications
• EvoJAX: Hardware-Accelerated Neuroevolution
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Meta-learning curiosity algorithms
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Ecological Reinforcement Learning
• Few-shot Quality-Diversity Optimization
• A Re-classification of Information Seeking Tasks and Their Computational Solutions
• On the Measure of Intelligence
• Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft
• Curriculum in Gradient-Based Meta-Reinforcement Learning
• Grounding Artificial Intelligence in the Origins of Human Behavior
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• Reinforcement Learning with Adaptive Curriculum Dynamics Randomization for Fault-Tolerant Robot Control
• Meta Automatic Curriculum Learning
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills
• Adversarial Environment Generation for Learning to Navigate the Web
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Learning Synthetic Environments for Reinforcement Learning with Evolution Strategies
• Simple Sensor Intentions for Exploration
• EVO-RL: Evolutionary-Driven Reinforcement Learning
• Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• First-Order Preconditioning via Hypergradient Descent
• An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
• Replay-Guided Adversarial Environment Design
• A Markov Decision Process Approach to Active Meta Learning
• AMaze: An intuitive benchmark generator for fast prototyping of generalizable agents
• Automatic Curricula via Expert Demonstrations
• A Monte Carlo Approach to Closing the Reality Gap
• L2E: Learning to Exploit Your Opponent
• Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning
• Divergent Search for Few-Shot Image Classification
• Follow the Neurally-Perturbed Leader for Adversarial Training