Strategic Attentive Writer for Learning Macro-Actions

Abstract

We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub- sequences by learning for how long the plan can be committed to - i.e. followed without re-planing. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro- actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.

References in corpus (5)

• Auto-Encoding Variational Bayes
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• End-to-End Training of Deep Visuomotor Policies
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft

Cited by in corpus (35)

• A Brief Survey of Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• State Representation Learning for Control: An Overview
• Hierarchical Multiscale Recurrent Neural Networks
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Variational Intrinsic Control
• A Survey of Deep Reinforcement Learning in Video Games
• Towards Deep Symbolic Reinforcement Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Neural Episodic Control
• Learning model-based planning from scratch
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Value Prediction Network
• Hierarchical Reinforcement Learning with Hindsight
• Learning Abstract Options
• AI Research Considerations for Human Existential Safety (ARCHES)
• Safe Option-Critic: Learning Safety in the Option-Critic Architecture
• Modular Deep Reinforcement Learning with Temporal Logic Specifications
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning
• Plan, Attend, Generate: Planning for Sequence-to-Sequence Models
• Learning to Multi-Task by Active Sampling
• TempoRL: Learning When to Act
• Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions
• Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder
• Online Baum-Welch algorithm for Hierarchical Imitation Learning
• Diversity-Driven Extensible Hierarchical Reinforcement Learning
• Hierarchical Reinforcement Learning with Deep Nested Agents
• Options Discovery with Budgeted Reinforcement Learning
• Plan, Attend, Generate: Character-level Neural Machine Translation with Planning in the Decoder
• Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model
• Autonomous Industrial Management via Reinforcement Learning: Self-Learning Agents for Decision-Making -- A Review
• Disentangling Options with Hellinger Distance Regularizer