Loss is its own Reward: Self-Supervision for Reinforcement Learning

Abstract

Reinforcement learning optimizes policies for expected cumulative reward. Need the supervision be so narrow? Reward is delayed and sparse for many tasks, making it a difficult and impoverished signal for end-to-end optimization. To augment reward, we consider a range of self-supervised tasks that incorporate states, actions, and successors to provide auxiliary losses. These losses offer ubiquitous and instantaneous supervision for representation learning even in the absence of reward. While current results show that learning from reward alone is feasible, pure reinforcement learning methods are constrained by computational and data efficiency issues that can be remedied by auxiliary losses. Self-supervised pre-training and joint optimization improve the data efficiency and policy returns of end-to-end reinforcement learning.

Cited by in corpus (14)

• Learning Action Representations for Reinforcement Learning
• Return-Based Contrastive Representation Learning for Reinforcement Learning
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• C-Learning: Learning to Achieve Goals via Recursive Classification
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Pretrained Encoders are All You Need
• Extracting Latent State Representations with Linear Dynamics from Rich Observations
• Work in Progress: Temporally Extended Auxiliary Tasks
• Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments
• Multitasking Inhibits Semantic Drift