Deep Reinforcement Learning With Macro-Actions

Abstract

Deep reinforcement learning has been shown to be a powerful framework for learning policies from complex high-dimensional sensory inputs to actions in complex tasks, such as the Atari domain. In this paper, we explore output representation modeling in the form of temporal abstraction to improve convergence and reliability of deep reinforcement learning approaches. We concentrate on macro-actions, and evaluate these on different Atari 2600 games, where we show that they yield significant improvements in learning speed. Additionally, we show that they can even achieve better scores than DQN. We offer analysis and explanation for both convergence and final results, revealing a problem deep RL approaches have with sparse reward signals.

References in corpus (8)

• Playing Atari with Deep Reinforcement Learning
• Prioritized Experience Replay
• Asynchronous Methods for Deep Reinforcement Learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Deep Exploration via Bootstrapped DQN
• Massively Parallel Methods for Deep Reinforcement Learning
• Hierarchical Solution of Markov Decision Processes using Macro-actions
• Classifying Options for Deep Reinforcement Learning

Cited by in corpus (6)

• Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes
• An Analysis of Frame-skipping in Reinforcement Learning
• Two-dimensional Anti-jamming Mobile Communication Based on Reinforcement Learning
• Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder
• DRL: Deep Reinforcement Learning for Intelligent Robot Control -- Concept, Literature, and Future
• Reusability and Transferability of Macro Actions for Reinforcement Learning