Return-Based Contrastive Representation Learning for Reinforcement Learning

Abstract

Recently, various auxiliary tasks have been proposed to accelerate representation learning and improve sample efficiency in deep reinforcement learning (RL). However, existing auxiliary tasks do not take the characteristics of RL problems into consideration and are unsupervised. By leveraging returns, the most important feedback signals in RL, we propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns. Our auxiliary loss is theoretically justified to learn representations that capture the structure of a new form of state-action abstraction, under which state-action pairs with similar return distributions are aggregated together. In low data regime, our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite, and achieves even better performance when combined with existing auxiliary tasks.

ICLR 2021

References in corpus (16)

• Model-Based Reinforcement Learning for Atari
• Learning to Navigate in Complex Environments
• Learning Latent Dynamics for Planning from Pixels
• When to use parametric models in reinforcement learning?
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Discovery of Useful Questions as Auxiliary Tasks
• Many-Goals Reinforcement Learning
• Universal Successor Features Approximators
• Deep Reinforcement and InfoMax Learning
• Predictive Information Accelerates Learning in RL
• Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcing
• Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning

Cited by in corpus (6)

• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• Domain Adaptation of Reinforcement Learning Agents based on Network Service Proximity
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Enhance Exploration in Safe Reinforcement Learning with Contrastive Representation Learning