APS: Active Pretraining with Successor Features

Abstract

We introduce a new unsupervised pretraining objective for reinforcement learning. During the unsupervised reward-free pretraining phase, the agent maximizes mutual information between tasks and states induced by the policy. Our key contribution is a novel lower bound of this intractable quantity. We show that by reinterpreting and combining variational successor features~\citep{Hansen2020Fast} with nonparametric entropy maximization~\citep{liu2021behavior}, the intractable mutual information can be efficiently optimized. The proposed method Active Pretraining with Successor Feature (APS) explores the environment via nonparametric entropy maximization, and the explored data can be efficiently leveraged to learn behavior by variational successor features. APS addresses the limitations of existing mutual information maximization based and entropy maximization based unsupervised RL, and combines the best of both worlds. When evaluated on the Atari 100k data-efficiency benchmark, our approach significantly outperforms previous methods combining unsupervised pretraining with task-specific finetuning.

Appeared in ICML 2021

Cited by in corpus (6)

• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Behavior From the Void: Unsupervised Active Pre-Training
• A First-Occupancy Representation for Reinforcement Learning
• Temporal Abstraction in Reinforcement Learning with the Successor Representation
• Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching