Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model

Abstract

High-dimensional state and action spaces combined with sparse reward structures in reinforcement learning (RL) environments typically require advanced control architectures. Hierarchical Reinforcement Learning (HRL) demonstrates superior performance compared to atomic RL approaches in these challenging scenarios. HRL can manage the complexity of commands to achieve task objectives through its hierarchical structure. One of the key challenges in HRL is efficiently training each level's policy with optimal data collection from its experience. Off-policy correction is a critical technique for facilitating sample-efficient off-policy training in HRL, as it addresses the non-stationary issue of higher-level policy training. However, existing methods typically employ indirect probabilistic approaches that fail to accurately capture the current capability of the lower-level policy. This mismatch ultimately constrains the effectiveness of higher-level policy training. In this paper, we propose a novel HRL model that supports direct off-policy correction based on a Flow-based Deep Generative Model (FDGM). This approach leverages the inverse operation of FDGM to achieve goals aligned with the current knowledge of the lower-level policy. Additionally, our model addresses the limitations of FDGM to enable its effective use in HRL. Through comparative experiments on benchmark environments, our model demonstrates superior performance over existing models.

arXiv commentPublished in the Journal of Artificial Intelligence Research (JAIR), Volume 86, 2026. This version corresponds to the published JAIR article

References in corpus (20)

• Continuous control with deep reinforcement learning
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Playing Atari with Deep Reinforcement Learning
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Hindsight Experience Replay
• MADE: Masked Autoencoder for Distribution Estimation
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Masked Autoregressive Flow for Density Estimation
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Improving Variational Inference with Inverse Autoregressive Flow
• Strategic Attentive Writer for Learning Macro-Actions
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization
• f-VAEs: Improve VAEs with Conditional Flows