Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning

Abstract

Goal-conditioned hierarchical reinforcement learning (HRL) has shown promising results for solving complex and long-horizon RL tasks. However, the action space of high-level policy in the goal-conditioned HRL is often large, so it results in poor exploration, leading to inefficiency in training. In this paper, we present HIerarchical reinforcement learning Guided by Landmarks (HIGL), a novel framework for training a high-level policy with a reduced action space guided by landmarks, i.e., promising states to explore. The key component of HIGL is twofold: (a) sampling landmarks that are informative for exploration and (b) encouraging the high-level policy to generate a subgoal towards a selected landmark. For (a), we consider two criteria: coverage of the entire visited state space (i.e., dispersion of states) and novelty of states (i.e., prediction error of a state). For (b), we select a landmark as the very first landmark in the shortest path in a graph whose nodes are landmarks. Our experiments demonstrate that our framework outperforms prior-arts across a variety of control tasks, thanks to efficient exploration guided by landmarks.

arXiv commentAccepted to NeurIPS 2021

References in corpus (20)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• VIME: Variational Information Maximizing Exploration
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Provably Efficient Maximum Entropy Exploration
• Efficient Exploration via State Marginal Matching
• Learning Multi-Level Hierarchies with Hindsight
• Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration
• Composable Planning with Attributes
• Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards
• Mapping State Space using Landmarks for Universal Goal Reaching
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Sparse Graphical Memory for Robust Planning
• Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning