Universal Successor Features Approximators

Abstract

The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs). Another way to generalise to new tasks is to exploit structure in the RL problem itself. Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs). Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI. We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical benefits and transfer abilities on a large-scale domain in which the agent has to navigate in a first-person perspective three-dimensional environment.

References in corpus (10)

• Overcoming catastrophic forgetting in neural networks
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Hindsight Experience Replay
• Distral: Robust Multitask Reinforcement Learning
• Grounded Language Learning in a Simulated 3D World
• Learning and Transfer of Modulated Locomotor Controllers
• Deep Successor Reinforcement Learning
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments
• Universal Successor Representations for Transfer Reinforcement Learning

Cited by in corpus (10)

• Transfer Learning in Deep Reinforcement Learning: A Survey
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Fast Adaptation via Policy-Dynamics Value Functions
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Learning One Representation to Optimize All Rewards
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• Planning in Hierarchical Reinforcement Learning: Guarantees for Using Local Policies
• Successor Feature Neural Episodic Control
• Rapid Task-Solving in Novel Environments