Reinforcement Learning with Unsupervised Auxiliary Tasks

Abstract

Deep reinforcement learning agents have achieved state-of-the-art results by directly maximising cumulative reward. However, environments contain a much wider variety of possible training signals. In this paper, we introduce an agent that also maximises many other pseudo-reward functions simultaneously by reinforcement learning. All of these tasks share a common representation that, like unsupervised learning, continues to develop in the absence of extrinsic rewards. We also introduce a novel mechanism for focusing this representation upon extrinsic rewards, so that learning can rapidly adapt to the most relevant aspects of the actual task. Our agent significantly outperforms the previous state-of-the-art on Atari, averaging 880\% expert human performance, and a challenging suite of first-person, three-dimensional \emph{Labyrinth} tasks leading to a mean speedup in learning of 10$\times$ and averaging 87\% expert human performance on Labyrinth.

Cited by in corpus (17)

• A Brief Survey of Deep Reinforcement Learning
• Forecasting day-ahead electricity prices in Europe: the importance of considering market integration
• Benchmarking Classic and Learned Navigation in Complex 3D Environments
• Distilling Policy Distillation
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• A Geometric Perspective on Optimal Representations for Reinforcement Learning
• On Inductive Biases in Deep Reinforcement Learning
• Design of intentional backdoors in sequential models
• Policy Evaluation Networks
• Universal Successor Features for Transfer Reinforcement Learning
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Motion Perception in Reinforcement Learning with Dynamic Objects
• Deep Reinforcement Learning with Decorrelation
• Acceleration of Actor-Critic Deep Reinforcement Learning for Visual Grasping in Clutter by State Representation Learning Based on Disentanglement of a Raw Input Image
• Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation
• Training an Interactive Helper
• Learning Good Representation via Continuous Attention