Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Abstract

Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for its own sake rather than to directly solve problems. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical value functions, operating at different temporal scales, with intrinsically motivated deep reinforcement learning. A top-level value function learns a policy over intrinsic goals, and a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse, delayed feedback: (1) a complex discrete stochastic decision process, and (2) the classic ATARI game `Montezuma's Revenge'.

14 pages, 7 figures

References in corpus (14)

• Prioritized Experience Replay
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Convolutional Inverse Graphics Network
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Deep Exploration via Bootstrapped DQN
• Massively Parallel Methods for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Language Understanding for Text-based Games Using Deep Reinforcement Learning
• Understanding Visual Concepts with Continuation Learning
• Efficient inference in occlusion-aware generative models of images
• Learning Purposeful Behaviour in the Absence of Rewards
• Binding via Reconstruction Clustering

Cited by in corpus (174)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• An Introduction to Deep Reinforcement Learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• A Review of Deep Reinforcement Learning for Smart Building Energy Management
• Data-Efficient Hierarchical Reinforcement Learning
• Exploration by Random Network Distillation
• Modular Multitask Reinforcement Learning with Policy Sketches
• A Survey of Machine Learning for Computer Architecture and Systems
• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
• Hybrid Reward Architecture for Reinforcement Learning
• Variational Intrinsic Control
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Unsupervised Learning of Object Keypoints for Perception and Control
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• A survey on intrinsic motivation in reinforcement learning
• Learning to design from humans: Imitating human designers through deep learning
• Deep Successor Reinforcement Learning
• Strategic Attentive Writer for Learning Macro-Actions
• Eigenoption Discovery through the Deep Successor Representation
• Latent Space Policies for Hierarchical Reinforcement Learning
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Multi-Level Discovery of Deep Options
• Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes
• Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving
• Learning Improvement Heuristics for Solving Routing Problems
• Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning
• Playing Doom with SLAM-Augmented Deep Reinforcement Learning
• Effective Diversity in Population Based Reinforcement Learning
• Deep Reinforcement Learning of Cell Movement in the Early Stage of C. elegans Embryogenesis
• Learn to Design the Heuristics for Vehicle Routing Problem
• Hierarchical Reinforcement Learning with Hindsight
• Learning Abstract Options
• Composable Planning with Attributes
• ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
• Hindsight policy gradients
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning
• Beating Atari with Natural Language Guided Reinforcement Learning
• Mapping State Space using Landmarks for Universal Goal Reaching
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal Clustering
• Hierarchical Imitation and Reinforcement Learning
• Learning Purposeful Behaviour in the Absence of Rewards
• Safe Option-Critic: Learning Safety in the Option-Critic Architecture
• Modular Deep Reinforcement Learning with Temporal Logic Specifications
• Active Long Term Memory Networks
• Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion Control
• Video Captioning via Hierarchical Reinforcement Learning
• Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning
• A Data-Efficient Deep Learning Approach for Deployable Multimodal Social Robots
• Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
• Learning to Compose Skills
• Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement Learning
• Multi-agent Deep Reinforcement Learning with Extremely Noisy Observations
• Value Iteration Networks on Multiple Levels of Abstraction
• ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay
• PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems
• Autonomous development and learning in artificial intelligence and robotics: Scaling up deep learning to human--like learning
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• Dynamics-aware Embeddings
• Meta-learning curiosity algorithms
• Deep Reinforcement Learning for Dexterous Manipulation with Concept Networks
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
• Sub-policy Adaptation for Hierarchical Reinforcement Learning
• Visualizing Dynamics: from t-SNE to SEMI-MDPs
• Saliency-based Sequential Image Attention with Multiset Prediction
• CompILE: Compositional Imitation Learning and Execution
• Analyzing Knowledge Transfer in Deep Q-Networks for Autonomously Handling Multiple Intersections
• Deep Reinforcement Learning With Macro-Actions
• MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics
• Discovering Blind Spots in Reinforcement Learning
• Hierarchical Reinforcement Learning with Abductive Planning
• Maximum Mutation Reinforcement Learning for Scalable Control
• Extracting Action Sequences from Texts Based on Deep Reinforcement Learning
• ReinBo: Machine Learning pipeline search and configuration with Bayesian Optimization embedded Reinforcement Learning
• Noise-Robust End-to-End Quantum Control using Deep Autoregressive Policy Networks
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Learning Hierarchical Teaching Policies for Cooperative Agents
• Hindsight Trust Region Policy Optimization
• Learning Symbolic Rules for Interpretable Deep Reinforcement Learning
• On the Complexity of Exploration in Goal-Driven Navigation
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Separation of Concerns in Reinforcement Learning
• Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Control
• A Layered Architecture for Active Perception: Image Classification using Deep Reinforcement Learning
• Deep Abstract Q-Networks
• ToyArchitecture: Unsupervised Learning of Interpretable Models of the World
• SOAC: The Soft Option Actor-Critic Architecture
• Bayesian Relational Memory for Semantic Visual Navigation
• A differential Hebbian framework for biologically-plausible motor control
• Towards Lifelong Self-Supervision: A Deep Learning Direction for Robotics
• Learning to Sequence Robot Behaviors for Visual Navigation
• Idiosyncrasies and challenges of data driven learning in electronic trading
• Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
• Challenges of Context and Time in Reinforcement Learning: Introducing Space Fortress as a Benchmark
• Few-shots Parallel Algorithm Portfolio Construction via Co-evolution
• Hierarchical principles of embodied reinforcement learning: A review
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
• DiGrad: Multi-Task Reinforcement Learning with Shared Actions
• Discovery of Options via Meta-Learned Subgoals
• Automata-Guided Hierarchical Reinforcement Learning for Skill Composition
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Provable Hierarchical Imitation Learning via EM
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• Online Baum-Welch algorithm for Hierarchical Imitation Learning
• Disentangling causal effects for hierarchical reinforcement learning
• Correlation-aware Cooperative Multigroup Broadcast 360° Video Delivery Network: A Hierarchical Deep Reinforcement Learning Approach
• Hierarchically Decoupled Imitation for Morphological Transfer
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• Learning to Solve a Rubik's Cube with a Dexterous Hand
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• Solving Large-Scale Extensive-Form Network Security Games via Neural Fictitious Self-Play
• Danger-aware Adaptive Composition of DRL Agents for Self-navigation
• Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning
• Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning
• Deep Reinforcement Learning Discovers Internal Models
• Micro-Objective Learning : Accelerating Deep Reinforcement Learning through the Discovery of Continuous Subgoals
• Learning Adaptive Display Exposure for Real-Time Advertising
• From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problem-solving
• State Space Decomposition and Subgoal Creation for Transfer in Deep Reinforcement Learning
• Abstract Value Iteration for Hierarchical Reinforcement Learning
• Energy-based Surprise Minimization for Multi-Agent Value Factorization
• ViDA-MAN: Visual Dialog with Digital Humans
• Learning Robust Options by Conditional Value at Risk Optimization
• Where to Look: A Unified Attention Model for Visual Recognition with Reinforcement Learning
• SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning
• Generalization to Novel Objects using Prior Relational Knowledge
• Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning
• Hierarchical Reinforcement Learning with Deep Nested Agents
• Deep Reinforcement Learning for Playing 2.5D Fighting Games
• Compositional planning in Markov decision processes: Temporal abstraction meets generalized logic composition
• Accelerating E-Commerce Search Engine Ranking by Contextual Factor Selection
• Lift-the-flap: what, where and when for context reasoning
• TTR-Based Reward for Reinforcement Learning with Implicit Model Priors
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• Knowledge-Based Sequential Decision-Making Under Uncertainty
• Expanding Motor Skills through Relay Neural Networks
• Questions to Guide the Future of Artificial Intelligence Research
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Feudal Steering: Hierarchical Learning for Steering Angle Prediction
• Options Discovery with Budgeted Reinforcement Learning
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• Transferring Agent Behaviors from Videos via Motion GANs
• Uniform State Abstraction For Reinforcement Learning
• Delayed Rewards Calibration via Reward Empirical Sufficiency
• Follow the Neurally-Perturbed Leader for Adversarial Training
• Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving
• Learning to Interrupt: A Hierarchical Deep Reinforcement Learning Framework for Efficient Exploration
• Learning Transferable Concepts in Deep Reinforcement Learning
• HILONet: Hierarchical Imitation Learning from Non-Aligned Observations
• Training Transition Policies via Distribution Matching for Complex Tasks
• Learning Time-Sensitive Strategies in Space Fortress
• Hierarchical Reinforcement Learning: Approximating Optimal Discounted TSP Using Local Policies
• The staircase property: How hierarchical structure can guide deep learning
• Hyperbolic Embeddings for Learning Options in Hierarchical Reinforcement Learning
• Discovering hierarchies using Imitation Learning from hierarchy aware policies
• Reinforcement Learning with Intrinsic Affinity for Personalized Prosperity Management
• Generation Drawing/Grinding Trajectoy Based on Hierarchical CVAE
• Delegating via Quitting Games