Deep Successor Reinforcement Learning
arXiv:1606.02396
Abstract
Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor map. The successor map represents the expected future state occupancy from any given state and the reward predictor maps states to scalar rewards. The value function of a state can be computed as the inner product between the successor map and the reward weights. In this paper, we present DSR, which generalizes SR within an end-to-end deep reinforcement learning framework. DSR has several appealing properties including: increased sensitivity to distal reward changes due to factorization of reward and world dynamics, and the ability to extract bottleneck states (subgoals) given successor maps trained under a random policy. We show the efficacy of our approach on two diverse environments given raw pixel observations -- simple grid-world domains (MazeBase) and the Doom game engine.
10 pages, 6 figures
References in corpus (16)
- Prioritized Experience Replay
- Asynchronous Methods for Deep Reinforcement Learning
- DRAW: A Recurrent Neural Network For Image Generation
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- Deep Convolutional Inverse Graphics Network
- Deep Exploration via Bootstrapped DQN
- Massively Parallel Methods for Deep Reinforcement Learning
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
- MazeBase: A Sandbox for Learning from Games
- Efficient inference in occlusion-aware generative models of images
- Understanding Visual Concepts with Continuation Learning
- Learning Purposeful Behaviour in the Absence of Rewards
- Binding via Reconstruction Clustering
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Cited by in corpus (51)
- A Brief Survey of Deep Reinforcement Learning
- Learning by Playing - Solving Sparse Reward Tasks from Scratch
- Transfer Learning in Deep Reinforcement Learning: A Survey
- A survey on intrinsic motivation in reinforcement learning
- Eigenoption Discovery through the Deep Successor Representation
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
- Value Prediction Network
- Independently Controllable Features
- Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
- Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments
- Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
- A neurally plausible model learns successor representations in partially observable environments
- Universal Successor Features Approximators
- A Critical Investigation of Deep Reinforcement Learning for Navigation
- Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
- Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
- Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
- Universal Successor Representations for Transfer Reinforcement Learning
- Dynamics-aware Embeddings
- Shaping Belief States with Generative Environment Models for RL
- MRL: Mind-aware Multi-agent Management Reinforcement Learning
- Synthesized Policies for Transfer and Adaptation across Tasks and Environments
- Universal Successor Features for Transfer Reinforcement Learning
- Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
- Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision
- Offline Meta-Reinforcement Learning with Online Self-Supervision
- UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
- AKF-SR: Adaptive Kalman Filtering-based Successor Representation
- Plan2Vec: Unsupervised Representation Learning by Latent Plans
- An Efficient Transfer Learning Framework for Multiagent Reinforcement Learning
- Expert-augmented actor-critic for ViZDoom and Montezumas Revenge
- PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
- Action-Sufficient State Representation Learning for Control with Structural Constraints
- Temporal Difference Learning with Neural Networks - Study of the Leakage Propagation Problem
- Deep Inverse Q-learning with Constraints
- Distributional Reinforcement Learning for Multi-Dimensional Reward Functions
- Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
- Target Driven Visual Navigation with Hybrid Asynchronous Universal Successor Representations
- Simple Sensor Intentions for Exploration
- Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
- A First-Occupancy Representation for Reinforcement Learning
- A New Framework for Query Efficient Active Imitation Learning
- Micro-Objective Learning : Accelerating Deep Reinforcement Learning through the Discovery of Continuous Subgoals
- A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning
- MAME : Model-Agnostic Meta-Exploration
- Temporal Abstraction in Reinforcement Learning with the Successor Representation
- Unbiased Deep Reinforcement Learning: A General Training Framework for Existing and Future Algorithms
- Building Intelligent Autonomous Navigation Agents
- Deep Episodic Value Iteration for Model-based Meta-Reinforcement Learning
- Evolving Indoor Navigational Strategies Using Gated Recurrent Units In NEAT
- Autonomous Industrial Management via Reinforcement Learning: Self-Learning Agents for Decision-Making -- A Review