Skew-Fit: State-Covering Self-Supervised Reinforcement Learning

Abstract

Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing goal reaching performance together with the entropy of the goal distribution, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. We prove that, under regularity conditions, Skew-Fit converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that combining Skew-Fit for learning goal distributions with existing goal-reaching methods outperforms a variety of prior methods on open-sourced visual goal-reaching tasks. Moreover, we demonstrate that Skew-Fit enables a real-world robot to learn to open a door, entirely from scratch, from pixels, and without any manually-designed reward function.

ICML 2020. 8 pages, 8 figures; 9 pages appendix (6 additional figures)

References in corpus (18)

• Continuous control with deep reinforcement learning
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Exploration by Random Network Distillation
• Episodic Curiosity through Reachability
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Provably Efficient Maximum Entropy Exploration
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Unsupervised Meta-Learning for Reinforcement Learning
• Maximum Entropy-Regularized Multi-Goal Reinforcement Learning
• Curiosity-Driven Experience Prioritization via Density Estimation
• Self-supervised Learning of Image Embedding for Continuous Control
• Many-Goals Reinforcement Learning

Cited by in corpus (34)

• A survey on intrinsic motivation in reinforcement learning
• Efficient Exploration via State Marginal Matching
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• Simple and Effective VAE Training with Calibrated Decoders
• On Reward-Free Reinforcement Learning with Linear Function Approximation
• If MaxEnt RL is the Answer, What is the Question?
• Action and Perception as Divergence Minimization
• World Model as a Graph: Learning Latent Landmarks for Planning
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Contextual Imagined Goals for Self-Supervised Robotic Learning
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
• Curriculum in Gradient-Based Meta-Reinforcement Learning
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• Geometric Entropic Exploration
• Marginalized State Distribution Entropy Regularization in Policy Optimization
• Plan2Vec: Unsupervised Representation Learning by Latent Plans
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Exploration via Hindsight Goal Generation
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• Self-Supervised Euphemism Detection and Identification for Content Moderation
• Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery
• Touch-based Curiosity for Sparse-Reward Tasks
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning
• Learning Compositional Neural Programs for Continuous Control
• Understanding the World Through Action
• Progressive growing of self-organized hierarchical representations for exploration
• How do Offline Measures for Exploration in Reinforcement Learning behave?