Successor Features for Transfer in Reinforcement Learning

Abstract

Transfer in reinforcement learning refers to the notion that generalization should occur not only within a task but also across tasks. We propose a transfer framework for the scenario where the reward function changes between tasks but the environment's dynamics remain the same. Our approach rests on two key ideas: "successor features", a value function representation that decouples the dynamics of the environment from the rewards, and "generalized policy improvement", a generalization of dynamic programming's policy improvement operation that considers a set of policies rather than a single one. Put together, the two ideas lead to an approach that integrates seamlessly within the reinforcement learning framework and allows the free exchange of information across tasks. The proposed method also provides performance guarantees for the transferred policy even before any learning has taken place. We derive two theorems that set our approach in firm theoretical ground and present experiments that show that it successfully promotes transfer in practice, significantly outperforming alternative methods in a sequence of navigation tasks and in the control of a simulated robotic arm.

Published at NIPS 2017

References in corpus (1)

• Adam: A Method for Stochastic Optimization

Cited by in corpus (20)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Towards Deep Symbolic Reinforcement Learning
• Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning
• Decoupling Dynamics and Reward for Transfer Learning
• Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments
• Beyond Preferences in AI Alignment
• Self-supervised Learning of Image Embedding for Continuous Control
• A Practical Guide to Multi-Objective Reinforcement Learning and Planning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Discovering Blind Spots in Reinforcement Learning
• AKF-SR: Adaptive Kalman Filtering-based Successor Representation
• Learning Actionable Representations with Goal-Conditioned Policies
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Tuning the Weights: The Impact of Initial Matrix Configurations on Successor Features Learning Efficacy
• Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning
• Block Contextual MDPs for Continual Learning
• Faster Reinforcement Learning Using Active Simulators
• Hierarchy through Composition with Linearly Solvable Markov Decision Processes
• Decoupled Learning of Environment Characteristics for Safe Exploration