Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Abstract

Intrinsically motivated goal exploration algorithms enable machines to discover repertoires of policies that produce a diversity of effects in complex environments. These exploration algorithms have been shown to allow real world robots to acquire skills such as tool use in high-dimensional continuous state and action spaces. However, they have so far assumed that self-generated goals are sampled in a specifically engineered feature space, limiting their autonomy. In this work, we propose to use deep representation learning algorithms to learn an adequate goal space. This is a developmental 2-stage approach: first, in a perceptual learning stage, deep learning algorithms use passive raw sensor observations of world changes to learn a corresponding latent space; then goal exploration happens in a second stage by sampling goals in this latent space. We present experiments where a simulated robot arm interacts with an object, and we show that exploration algorithms using such learned representations can match the performance obtained using engineered representations.

References in corpus (5)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• VIME: Variational Information Maximizing Exploration
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Reverse Curriculum Generation for Reinforcement Learning
• Independently Controllable Factors

Cited by in corpus (18)

• Exploration in Deep Reinforcement Learning: A Survey
• State Representation Learning for Control: An Overview
• Episodic Curiosity through Reachability
• A survey on intrinsic motivation in reinforcement learning
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• From exploration to control: learning object manipulation skills through novelty search and local adaptation
• Computational Theories of Curiosity-Driven Learning
• Accuracy-based Curriculum Learning in Deep Reinforcement Learning
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
• Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires
• Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Autonomous discovery of the goal space to learn a parameterized skill
• DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• Policy Manifold Search for Improving Diversity-based Neuroevolution
• Progressive growing of self-organized hierarchical representations for exploration