Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching

Abstract

Learning meaningful behaviors in the absence of reward is a difficult problem in reinforcement learning. A desirable and challenging unsupervised objective is to learn a set of diverse skills that provide a thorough coverage of the state space while being directed, i.e., reliably reaching distinct regions of the environment. In this paper, we build on the mutual information framework for skill discovery and introduce UPSIDE, which addresses the coverage-directedness trade-off in the following ways: 1) We design policies with a decoupled structure of a directed skill, trained to reach a specific region, followed by a diffusing part that induces a local coverage. 2) We optimize policies by maximizing their number under the constraint that each of them reaches distinct regions of the environment (i.e., they are sufficiently discriminable) and prove that this serves as a lower bound to the original mutual information objective. 3) Finally, we compose the learned directed skills into a growing tree that adaptively covers the environment. We illustrate in several navigation and control environments how the skills learned by UPSIDE solve sparse-reward downstream tasks better than existing baselines.

arXiv commentICLR 2022

References in corpus (19)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• VIME: Variational Information Maximizing Exploration
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Variational Intrinsic Control
• Efficient Exploration via State Marginal Matching
• Variational Option Discovery Algorithms
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Eigenoption Discovery through the Deep Successor Representation
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• APS: Active Pretraining with Successor Features
• Discovering Options for Exploration by Minimizing Cover Time
• Reset-Free Lifelong Learning with Skill-Space Planning
• Relative Variational Intrinsic Control
• Learning more skills through optimistic exploration
• Continual Learning of Control Primitives: Skill Discovery via Reset-Games
• IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL
• SaLinA: Sequential Learning of Agents