Value Prediction Network

Abstract

This paper proposes a novel deep reinforcement learning (RL) architecture, called Value Prediction Network (VPN), which integrates model-free and model-based RL methods into a single neural network. In contrast to typical model-based RL methods, VPN learns a dynamics model whose abstract states are trained to make option-conditional predictions of future values (discounted sum of rewards) rather than of future observations. Our experimental results show that VPN has several advantages over both model-free and model-based baselines in a stochastic environment where careful planning is required but building an accurate observation-prediction model is difficult. Furthermore, VPN outperforms Deep Q-Network (DQN) on several Atari games even with short-lookahead planning, demonstrating its potential as a new way of learning a good state representation.

NIPS 2017

References in corpus (16)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Dueling Network Architectures for Deep Reinforcement Learning
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Learning Continuous Control Policies by Stochastic Value Gradients
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Unsupervised Learning for Physical Interaction through Video Prediction
• Control of Memory, Active Perception, and Action in Minecraft
• The Predictron: End-To-End Learning and Planning
• Recurrent Environment Simulators
• Neural Map: Structured Memory for Deep Reinforcement Learning
• Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping
• Deep Successor Reinforcement Learning
• Strategic Attentive Writer for Learning Macro-Actions
• Prediction and Control with Temporal Segment Models

Cited by in corpus (24)

• An Introduction to Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Investigating Human Priors for Playing Video Games
• Learning Plannable Representations with Causal InfoGAN
• Universal Planning Networks
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search
• Value Propagation Networks
• Parallel Exploration via Negatively Correlated Search
• Efficient reinforcement learning control for continuum robots based on Inexplicit Prior Knowledge
• Contrastive Variational Reinforcement Learning for Complex Observations
• Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
• Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks
• Gated Path Planning Networks
• Integrating Algorithmic Planning and Deep Learning for Partially Observable Navigation
• ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search
• Critical Percolation as a Framework to Analyze the Training of Deep Networks
• Self-Consistent Models and Values
• Transfer Value Iteration Networks
• Learning the Reward Function for a Misspecified Model