Loss is its own Reward: Self-Supervision for Reinforcement Learning

Abstract

Reinforcement learning optimizes policies for expected cumulative reward. Need the supervision be so narrow? Reward is delayed and sparse for many tasks, making it a difficult and impoverished signal for end-to-end optimization. To augment reward, we consider a range of self-supervised tasks that incorporate states, actions, and successors to provide auxiliary losses. These losses offer ubiquitous and instantaneous supervision for representation learning even in the absence of reward. While current results show that learning from reward alone is feasible, pure reinforcement learning methods are constrained by computational and data efficiency issues that can be remedied by auxiliary losses. Self-supervised pre-training and joint optimization improve the data efficiency and policy returns of end-to-end reinforcement learning.

References in corpus (8)

• Auto-Encoding Variational Bayes
• Playing Atari with Deep Reinforcement Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Asynchronous Methods for Deep Reinforcement Learning
• Adversarial Feature Learning
• VIME: Variational Information Maximizing Exploration
• Unsupervised Visual Representation Learning by Context Prediction
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics

Cited by in corpus (42)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• State Representation Learning for Control: An Overview
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Unsupervised Learning via Meta-Learning
• Reinforcement Learning from Imperfect Demonstrations
• A survey on intrinsic motivation in reinforcement learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Contingency-Aware Exploration in Reinforcement Learning
• A Deep Policy Inference Q-Network for Multi-Agent Systems
• Many-Goals Reinforcement Learning
• Multi-task Deep Reinforcement Learning with PopArt
• Predictive Information Accelerates Learning in RL
• Learning Action Representations for Reinforcement Learning
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• See, Hear, Explore: Curiosity via Audio-Visual Association
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Auxiliary Task Reweighting for Minimum-data Learning
• Learning Actionable Representations from Visual Observations
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Exploratory State Representation Learning
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Using Monte Carlo Tree Search as a Demonstrator within Asynchronous Deep RL
• Learning Markov State Abstractions for Deep Reinforcement Learning
• Action-Based Representation Learning for Autonomous Driving
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Belief-Grounded Networks for Accelerated Robot Learning under Partial Observability
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Pretrained Encoders are All You Need
• Extracting Latent State Representations with Linear Dynamics from Rich Observations
• Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments
• Learning of feature points without additional supervision improves reinforcement learning from images
• Multitasking Inhibits Semantic Drift
• Work in Progress: Temporally Extended Auxiliary Tasks
• State Representation Learning from Demonstration