Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning
arXiv:2002.05954 · doi:10.1109/LRA.2021.3060403
Abstract
We present HiDe, a novel hierarchical reinforcement learning architecture that successfully solves long horizon control tasks and generalizes to unseen test scenarios. Functional decomposition between planning and low-level control is achieved by explicitly separating the state-action spaces across the hierarchy, which allows the integration of task-relevant knowledge per layer. We propose an RL-based planner to efficiently leverage the information in the planning layer of the hierarchy, while the control layer learns a goal-conditioned control policy. The hierarchy is trained jointly but allows for the modular transfer of policy layers across hierarchies of different agents. We experimentally show that our method generalizes across unseen test environments and can scale to 3x horizon length compared to both learning and non-learning based methods. We evaluate on complex continuous control tasks with sparse rewards, including navigation and robot manipulation.
References in corpus (13)
- DeepMind Control Suite
- FeUdal Networks for Hierarchical Reinforcement Learning
- Data-Efficient Hierarchical Reinforcement Learning
- Benchmarking Model-Based Reinforcement Learning
- Driving Policy Transfer via Modularity and Abstraction
- Universal Planning Networks
- Learning Multi-Level Hierarchies with Hindsight
- Planning with Goal-Conditioned Policies
- Demonstration-Guided Deep Reinforcement Learning of Control Policies for Dexterous Human-Robot Interaction
- Improved Learning of Robot Manipulation Tasks via Tactile Intrinsic Motivation
- Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
- Value Propagation Networks
- Hierarchically Decoupled Imitation for Morphological Transfer