Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Abstract

Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

arXiv commentPresented as an oral at the NeurIPS 2019 DeepRL Workshop

References in corpus (15)

• Continuous control with deep reinforcement learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Emergence of Locomotion Behaviours in Rich Environments
• Noisy Networks for Exploration
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Parameter Space Noise for Exploration
• Data-Efficient Hierarchical Reinforcement Learning
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Meta Learning Shared Hierarchies
• Generalization and Exploration via Randomized Value Functions
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Eigenoption Discovery through the Deep Successor Representation
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

Cited by in corpus (19)

• A survey on intrinsic motivation in reinforcement learning
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Active Predictive Coding: A Unified Neural Framework for Learning Hierarchical World Models for Perception and Planning
• Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Behavior Priors for Efficient Reinforcement Learning
• Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
• Reset-Free Lifelong Learning with Skill-Space Planning
• From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agent
• A Survey of Exploration Methods in Reinforcement Learning
• Discovery of Options via Meta-Learned Subgoals
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• Hierarchical Skills for Efficient Exploration
• Active Hierarchical Imitation and Reinforcement Learning
• Provable Hierarchy-Based Meta-Reinforcement Learning
• Learning Transferable Concepts in Deep Reinforcement Learning
• TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution