Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes

Abstract

In recent years, reinforcement learning has achieved many remarkable successes due to the growing adoption of deep learning techniques and the rapid growth in computing power. Nevertheless, it is well-known that flat reinforcement learning algorithms are often not able to learn well and data-efficient in tasks having hierarchical structures, e.g. consisting of multiple subtasks. Hierarchical reinforcement learning is a principled approach that is able to tackle these challenging tasks. On the other hand, many real-world tasks usually have only partial observability in which state measurements are often imperfect and partially observable. The problems of RL in such settings can be formulated as a partially observable Markov decision process (POMDP). In this paper, we study hierarchical RL in POMDP in which the tasks have only partial observability and possess hierarchical properties. We propose a hierarchical deep reinforcement learning approach for learning in hierarchical POMDP. The deep hierarchical RL algorithm is proposed to apply to both MDP and POMDP learning. We evaluate the proposed algorithm on various challenging hierarchical POMDP.

arXiv commentThis work has been submitted to the IEEE for possible publication

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Evolutionary Algorithms for Reinforcement Learning
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Multi-Level Discovery of Deep Options
• MazeBase: A Sandbox for Learning from Games
• Classifying Options for Deep Reinforcement Learning
• Deep Reinforcement Learning With Macro-Actions
• Micro-Objective Learning : Accelerating Deep Reinforcement Learning through the Discovery of Continuous Subgoals

Cited by in corpus (1)

• Learning What to Memorize: Using Intrinsic Motivation to Form Useful Memory in Partially Observable Reinforcement Learning