Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

Abstract

Real-world tasks are often highly structured. Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL). However, identifying the hierarchical policy structure that enhances the performance of RL is not a trivial task. In this paper, we propose an HRL method that learns a latent variable of a hierarchical policy using mutual information maximization. Our approach can be interpreted as a way to learn a discrete and latent representation of the state-action space. To learn option policies that correspond to modes of the advantage function, we introduce advantage-weighted importance sampling. In our HRL method, the gating policy learns to select option policies based on an option-value function, and these option policies are optimized based on the deterministic policy gradient method. This framework is derived by leveraging the analogy between a monolithic policy in standard RL and a hierarchical policy in HRL by using a deterministic option policy. Experimental results indicate that our HRL approach can learn a diversity of options and that it can enhance the performance of RL in continuous control tasks.

arXiv comment16 pages, ICLR 2019

References in corpus (5)

• Deep Reinforcement Learning that Matters
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• FeUdal Networks for Hierarchical Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Hierarchical Policy Search via Return-Weighted Density Estimation

Cited by in corpus (8)

• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• Hierarchical Reinforcement Learning for Quadruped Locomotion
• MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics
• SOAC: The Soft Option Actor-Critic Architecture
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• InfoRL: Interpretable Reinforcement Learning using Information Maximization
• Learning Transferable Concepts in Deep Reinforcement Learning
• Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model