Automated curricula through setter-solver interactions

Abstract

Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic or sparsely rewarding environments these correlations are often too small, or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula--the breakdown of tasks into simpler, static challenges with dense rewards--to build up to complex behaviors. While curricula are also useful for artificial agents, hand-crafting them is time consuming. This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich, dynamic environments. Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage to construct useful curricula. We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent is tasked to achieve a single goal selected from a set of possible goals that varies between episodes, and identify challenges for future work. Finally, we demonstrate the value of a novel technique that guides agents towards a desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex, otherwise unlearnable goals, and to our knowledge are the first to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary between episodes.

References in corpus (15)

• Conditional Generative Adversarial Nets
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Training Very Deep Networks
• Density estimation using Real NVP
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Large-Scale Study of Curiosity-Driven Learning
• Visual Reinforcement Learning with Imagined Goals
• Learning to Execute
• Variational Intrinsic Control
• Automatic Goal Generation for Reinforcement Learning Agents
• Classification Accuracy Score for Conditional Generative Models
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling

Cited by in corpus (11)

• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• Meta Automatic Curriculum Learning
• Self-Organizing Intelligent Matter: A blueprint for an AI generating algorithm
• HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents
• Training a First-Order Theorem Prover from Synthetic Data
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• Biological Blueprints for Next Generation AI Systems
• Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning