Reinforcement Learning with Unsupervised Auxiliary Tasks

Abstract

Deep reinforcement learning agents have achieved state-of-the-art results by directly maximising cumulative reward. However, environments contain a much wider variety of possible training signals. In this paper, we introduce an agent that also maximises many other pseudo-reward functions simultaneously by reinforcement learning. All of these tasks share a common representation that, like unsupervised learning, continues to develop in the absence of extrinsic rewards. We also introduce a novel mechanism for focusing this representation upon extrinsic rewards, so that learning can rapidly adapt to the most relevant aspects of the actual task. Our agent significantly outperforms the previous state-of-the-art on Atari, averaging 880\% expert human performance, and a challenging suite of first-person, three-dimensional \emph{Labyrinth} tasks leading to a mean speedup in learning of 10$\times$ and averaging 87\% expert human performance on Labyrinth.

Cited by in corpus (105)

• A Brief Survey of Deep Reinforcement Learning
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach
• Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey
• Forecasting day-ahead electricity prices in Europe: the importance of considering market integration
• First return, then explore
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• Deep Reinforcement Learning (DRL): Another Perspective for Unsupervised Wireless Localization
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Self-Supervised Generalisation with Meta Auxiliary Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Decoupling Representation Learning from Reinforcement Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Contingency-Aware Exploration in Reinforcement Learning
• Open-Ended Learning Leads to Generally Capable Agents
• Chasing Ghosts: Instruction Following as Bayesian State Tracking
• Dealing with Sparse Rewards in Reinforcement Learning
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Benchmarking Classic and Learned Navigation in Complex 3D Environments
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Distilling Policy Distillation
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• A Geometric Perspective on Optimal Representations for Reinforcement Learning
• Operationally meaningful representations of physical systems in neural networks
• Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning
• Interactive Fiction Games: A Colossal Adventure
• Predictive Information Accelerates Learning in RL
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning
• General Value Function Networks
• On Inductive Biases in Deep Reinforcement Learning
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
• Design of intentional backdoors in sequential models
• See, Hear, Explore: Curiosity via Audio-Visual Association
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Policy Evaluation Networks
• Remember and Forget for Experience Replay
• Deep Learning for Embodied Vision Navigation: A Survey
• Muesli: Combining Improvements in Policy Optimization
• Reward Estimation for Variance Reduction in Deep Reinforcement Learning
• Universal Successor Features for Transfer Reinforcement Learning
• A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning
• Training Larger Networks for Deep Reinforcement Learning
• Motion Perception in Reinforcement Learning with Dynamic Objects
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• Learning predictive representations in autonomous driving to improve deep reinforcement learning
• Model-Based Opponent Modeling
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Policy Search with Rare Significant Events: Choosing the Right Partner to Cooperate with
• Deep Reinforcement Learning with Decorrelation
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Acceleration of Actor-Critic Deep Reinforcement Learning for Visual Grasping in Clutter by State Representation Learning Based on Disentanglement of a Raw Input Image
• Bayesian Relational Memory for Semantic Visual Navigation
• Exploratory State Representation Learning
• Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• XLVIN: eXecuted Latent Value Iteration Nets
• Positional Encoder Graph Neural Networks for Geographic Data
• Towards mental time travel: a hierarchical memory for reinforcement learning agents
• Maximizing Information Gain in Partially Observable Environments via Prediction Reward
• Active Perception and Representation for Robotic Manipulation
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Importance Resampling for Off-policy Prediction
• Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains
• Scene-Intuitive Agent for Remote Embodied Visual Grounding
• Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization
• Bridging the Imitation Gap by Adaptive Insubordination
• Vision-Dialog Navigation by Exploring Cross-modal Memory
• SALT: Subspace Alignment as an Auxiliary Learning Task for Domain Adaptation
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Control What You Can: Intrinsically Motivated Task-Planning Agent
• Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation
• Learning to Sample with Local and Global Contexts in Experience Replay Buffer
• A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning
• Sequential Learning for Domain Generalization
• Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships
• The Traveling Observer Model: Multi-task Learning Through Spatial Variable Embeddings
• Taxonomy Completion via Triplet Matching Network
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• Value-driven Hindsight Modelling
• Improving RNN Transducer Based ASR with Auxiliary Tasks
• Temporal Abstraction in Reinforcement Learning with the Successor Representation
• Recognizing Object Affordances to Support Scene Reasoning for Manipulation Tasks
• Towards Target-Driven Visual Navigation in Indoor Scenes via Generative Imitation Learning
• The Pitfall of More Powerful Autoencoders in Lidar-Based Navigation
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning
• Unsupervised Salient Patch Selection for Data-Efficient Reinforcement Learning
• A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
• Training an Interactive Helper
• Learning Good Representation via Continuous Attention
• Learning State Representations from Random Deep Action-conditional Predictions
• Conditional Importance Sampling for Off-Policy Learning
• Work in Progress: Temporally Extended Auxiliary Tasks