Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills

Abstract

Acquiring abilities in the absence of a task-oriented reward function is at the frontier of reinforcement learning research. This problem has been studied through the lens of empowerment, which draws a connection between option discovery and information theory. Information-theoretic skill discovery methods have garnered much interest from the community, but little research has been conducted in understanding their limitations. Through theoretical analysis and empirical evidence, we show that existing algorithms suffer from a common limitation -- they discover options that provide a poor coverage of the state space. In light of this, we propose 'Explore, Discover and Learn' (EDL), an alternative approach to information-theoretic skill discovery. Crucially, EDL optimizes the same information-theoretic objective derived from the empowerment literature, but addresses the optimization problem using different machinery. We perform an extensive evaluation of skill discovery methods on controlled environments and show that EDL offers significant advantages, such as overcoming the coverage problem, reducing the dependence of learned skills on the initial state, and allowing the user to define a prior over which behaviors should be learned. Code is publicly available at https://github.com/victorcampos7/edl.

arXiv comment17 pages, 11 figures. Code is publicly available at https://github.com/victorcampos7/edl

References in corpus (36)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Neural Discrete Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Robots that can adapt like animals
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Solving Rubik's Cube with a Robot Hand
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• VIME: Variational Information Maximizing Exploration
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Illuminating search spaces by mapping elites
• Reinforcement Learning with Deep Energy-Based Policies
• Hindsight Experience Replay
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Exploration by Random Network Distillation
• Unifying Count-Based Exploration and Intrinsic Motivation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Variational Intrinsic Control
• Self-Imitation Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Meta Learning Shared Hierarchies
• Provably Efficient Maximum Entropy Exploration
• Efficient Exploration via State Marginal Matching
• Variational Option Discovery Algorithms
• Dynamics-Aware Unsupervised Discovery of Skills
• Multi-Level Discovery of Deep Options
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Self-supervised Learning of Image Embedding for Continuous Control
• Learning Latent Plans from Play

Cited by in corpus (4)

• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• Learning Task Agnostic Skills with Data-driven Guidance