Curiosity Driven Exploration of Learned Disentangled Goal Spaces

Abstract

Intrinsically motivated goal exploration processes enable agents to autonomously sample goals to explore efficiently complex environments with high-dimensional continuous actions. They have been applied successfully to real world robots to discover repertoires of policies producing a wide diversity of effects. Often these algorithms relied on engineered goal spaces but it was recently shown that one can use deep representation learning algorithms to learn an adequate goal space in simple environments. However, in the case of more complex environments containing multiple objects or distractors, an efficient exploration requires that the structure of the goal space reflects the one of the environment. In this paper we show that using a disentangled goal space leads to better exploration performances than an entangled goal space. We further show that when the representation is disentangled, one can leverage it by sampling goals that maximize learning progress in a modular manner. Finally, we show that the measure of learning progress, used to drive curiosity-driven exploration, can be used simultaneously to discover abstract independently controllable features of the environment.

arXiv commentThe code used in the experiments is available at https://github.com/flowersteam/Curiosity_Driven_Goal_Exploration

References in corpus (12)

• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Robots that can adapt like animals
• VIME: Variational Information Maximizing Exploration
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Unifying Count-Based Exploration and Intrinsic Motivation
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Reverse Curriculum Generation for Reinforcement Learning
• Early Visual Concept Learning with Unsupervised Deep Learning
• Representation Learning: A Review and New Perspectives
• Independently Controllable Factors
• Disentangling the independently controllable factors of variation by interacting with the world
• Computational Theories of Curiosity-Driven Learning

Cited by in corpus (29)

• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• A survey on intrinsic motivation in reinforcement learning
• Contrastive Learning of Structured World Models
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reduction
• Disentangling Factors of Variation Using Few Labels
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• From exploration to control: learning object manipulation skills through novelty search and local adaptation
• Curious iLQR: Resolving Uncertainty in Model-based RL
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• Hierarchical Quality-Diversity for Online Damage Recovery
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• On the Fairness of Disentangled Representations
• Learning Actionable Representations with Goal-Conditioned Policies
• DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• Disentangled Cumulants Help Successor Representations Transfer to New Tasks
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Learning Independently-Obtainable Reward Functions
• Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards
• Adversarial Active Exploration for Inverse Dynamics Model Learning
• Disentangled (Un)Controllable Features
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Learning in Sparse Rewards settings through Quality-Diversity algorithms
• Towards Better Understanding of Disentangled Representations via Mutual Information