Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Abstract

Hierarchical Reinforcement Learning (HRL) is a promising approach to solving long-horizon problems with sparse and delayed rewards. Many existing HRL algorithms either use pre-trained low-level skills that are unadaptable, or require domain-specific information to define low-level rewards. In this paper, we aim to adapt low-level skills to downstream tasks while maintaining the generality of reward design. We propose an HRL framework which sets auxiliary rewards for low-level skill training based on the advantage function of the high-level policy. This auxiliary reward enables efficient, simultaneous learning of the high-level policy and low-level skills without using task-specific knowledge. In addition, we also theoretically prove that optimizing low-level skills with this auxiliary reward will increase the task return for the joint policy. Experimental results show that our algorithm dramatically outperforms other state-of-the-art HRL methods in Mujoco domains. We also find both low-level and high-level policies trained by our algorithm transferable.

Camera ready version for NeurIPS 2019

Cited by in corpus (11)

• Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• SOAC: The Soft Option Actor-Critic Architecture
• Hierarchically Decoupled Imitation for Morphological Transfer
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
• Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learning
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Lifetime policy reuse and the importance of task capacity
• Reward Shaping with Dynamic Trajectory Aggregation
• Reward Shaping with Subgoals for Social Navigation