Sub-policy Adaptation for Hierarchical Reinforcement Learning
arXiv:1906.05862
Abstract
Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new hierarchical policy gradient with an unbiased latent-dependent baseline, and we introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy jointly. Second, we propose a method for training time-abstractions that improves the robustness of the obtained skills to environment changes. Code and results are available at sites.google.com/view/hippo-rl
arXiv commentICLR 2020
References in corpus (10)
- Trust Region Policy Optimization
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- Visual Reinforcement Learning with Imagined Goals
- FeUdal Networks for Hierarchical Reinforcement Learning
- Data-Efficient Hierarchical Reinforcement Learning
- Modular Multitask Reinforcement Learning with Policy Sketches
- Hierarchical visuomotor control of humanoids
- Hierarchical Reinforcement Learning with Hindsight
- Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information
- Self-supervised Learning of Image Embedding for Continuous Control