Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning

Abstract

Goal-conditioned hierarchical reinforcement learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is often large. Searching in a large goal space poses difficulties for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a $k$-step adjacent region of the current state using an adjacency constraint. We theoretically prove that the proposed adjacency constraint preserves the optimal hierarchical policy in deterministic MDPs, and show that this constraint can be practically implemented by training an adjacency network that can discriminate between adjacent and non-adjacent subgoals. Experimental results on discrete and continuous control tasks show that incorporating the adjacency constraint improves the performance of state-of-the-art HRL approaches in both deterministic and stochastic environments.

arXiv commentAccepted by NeurIPS 2020

References in corpus (17)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Visual Reinforcement Learning with Imagined Goals
• Data-Efficient Hierarchical Reinforcement Learning
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• Deep Successor Reinforcement Learning
• Learning Multi-Level Hierarchies with Hindsight
• Metrics for Finite Markov Decision Processes
• Planning with Goal-Conditioned Policies
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Composable Planning with Attributes
• What can I do here? A Theory of Affordances in Reinforcement Learning
• Mapping State Space using Landmarks for Universal Goal Reaching
• Q-Learning in enormous action spaces via amortized approximate maximization
• Self-supervised Learning of Image Embedding for Continuous Control

Cited by in corpus (2)

• Intelligent problem-solving as integrated hierarchical reinforcement learning
• Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis