Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

Abstract

Intrinsically motivated spontaneous exploration is a key enabler of autonomous developmental learning in human children. It enables the discovery of skill repertoires through autotelic learning, i.e. the self-generation, self-selection, self-ordering and self-experimentation of learning goals. We present an algorithmic approach called Intrinsically Motivated Goal Exploration Processes (IMGEP) to enable similar properties of autonomous learning in machines. The IMGEP architecture relies on several principles: 1) self-generation of goals, generalized as parameterized fitness functions; 2) selection of goals based on intrinsic rewards; 3) exploration with incremental goal-parameterized policy search and exploitation with a batch learning algorithm; 4) systematic reuse of information acquired when targeting a goal for improving towards other goals. We present a particularly efficient form of IMGEP, called AMB, that uses a population-based policy and an object-centered spatio-temporal modularity. We provide several implementations of this architecture and demonstrate their ability to automatically generate a learning curriculum within several experimental setups. One of these experiments includes a real humanoid robot exploring multiple spaces of goals with several hundred continuous dimensions and with distractors. While no particular target goal is provided to these autotelic agents, this curriculum allows the discovery of diverse skills that act as stepping stones for learning more complex skills, e.g. nested tool use.

arXiv commentAccepted at JMLR 2022

References in corpus (12)

• Robots that can adapt like animals
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• VIME: Variational Information Maximizing Exploration
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Illuminating search spaces by mapping elites
• Visual Reinforcement Learning with Imagined Goals
• Learning to Execute
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Variational Intrinsic Control
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Automated Curriculum Learning for Neural Networks

Cited by in corpus (65)

• Exploration in Deep Reinforcement Learning: A Survey
• Emergent Tool Use From Multi-Agent Autocurricula
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Go-Explore: a New Approach for Hard-Exploration Problems
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Small Sample Learning in Big Data Era
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Unsupervised Meta-Learning for Reinforcement Learning
• Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
• Goal-Directed Planning for Habituated Agents by Active Inference Using a Variational Recurrent Neural Network
• Intrinsic Motivation and Episodic Memories for Robot Exploration of High-Dimensional Sensory Spaces
• Computational Theories of Curiosity-Driven Learning
• Accuracy-based Curriculum Learning in Deep Reinforcement Learning
• Learning Latent Plans from Play
• Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• MAPPER: Multi-Agent Path Planning with Evolutionary Reinforcement Learning in Mixed Dynamic Environments
• Autonomous discovery of the goal space to learn a parameterized skill
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Few-shot Quality-Diversity Optimization
• Robots Learn Increasingly Complex Tasks with Intrinsic Motivation and Automatic Curriculum Learning
• Rethinking Continual Learning for Autonomous Agents and Robots
• Language Grounding through Social Interactions and Curiosity-Driven Multi-Goal Learning
• DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Meta Automatic Curriculum Learning
• Generating Automatic Curricula via Self-Supervised Active Domain Randomization
• A Survey of Exploration Methods in Reinforcement Learning
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Adversarial Intrinsic Motivation for Reinforcement Learning
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• Competitive Experience Replay
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Exploration via Hindsight Goal Generation
• Simple Sensor Intentions for Exploration
• Touch-based Curiosity for Sparse-Reward Tasks
• Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation
• From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solving
• Self-Emotion-Mediated Exploration in Artificial Intelligence Mirrors: Findings from Cognitive Psychology
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• Exploration of Self-Propelling Droplets Using a Curiosity Driven Robotic Assistant
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Autonomous Goal Exploration using Learned Goal Spaces for Visuomotor Skill Acquisition in Robots
• Homotopy Based Reinforcement Learning with Maximum Entropy for Autonomous Air Combat
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• Wasserstein Distance Maximizing Intrinsic Control
• Autonomous learning of multiple, context-dependent tasks
• Progressive growing of self-organized hierarchical representations for exploration
• Developing cooperative policies for multi-stage tasks
• Autonomous Open-Ended Learning of Interdependent Tasks
• What Can This Robot Do? Learning from Appearance and Experiments