Learning Multi-Level Hierarchies with Hindsight

Abstract

Hierarchical agents have the potential to solve sequential decision making tasks with greater sample efficiency than their non-hierarchical counterparts because hierarchical agents can break down tasks into sets of subtasks that only require short sequences of decisions. In order to realize this potential of faster learning, hierarchical agents need to be able to learn their multiple levels of policies in parallel so these simpler subproblems can be solved simultaneously. Yet, learning multiple levels of policies in parallel is hard because it is inherently unstable: changes in a policy at one level of the hierarchy may cause changes in the transition and reward functions at higher levels in the hierarchy, making it difficult to jointly learn multiple levels of policies. In this paper, we introduce a new Hierarchical Reinforcement Learning (HRL) framework, Hierarchical Actor-Critic (HAC), that can overcome the instability issues that arise when agents try to jointly learn multiple levels of policies. The main idea behind HAC is to train each level of the hierarchy independently of the lower levels by training each level as if the lower level policies are already optimal. We demonstrate experimentally in both grid world and simulated robotics domains that our approach can significantly accelerate learning relative to other non-hierarchical and hierarchical methods. Indeed, our framework is the first to successfully learn 3-level hierarchies in parallel in tasks with continuous state and action spaces.

ICLR 2019 Accepted Paper

Cited by in corpus (24)

• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Learning to Set Waypoints for Audio-Visual Navigation
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower
• HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators
• Demonstration-Guided Reinforcement Learning with Learned Skills
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• Hierarchical principles of embodied reinforcement learning: A review
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
• Hierarchies of Planning and Reinforcement Learning for Robot Navigation
• Exploration via Hindsight Goal Generation
• From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solving
• MGHRL: Meta Goal-generation for Hierarchical Reinforcement Learning
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• Provable Hierarchy-Based Meta-Reinforcement Learning
• Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
• Risk Conditioned Neural Motion Planning
• Lifetime policy reuse and the importance of task capacity