CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Abstract

We present CURL: Contrastive Unsupervised Representations for Reinforcement Learning. CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted features. CURL outperforms prior pixel-based methods, both model-based and model-free, on complex tasks in the DeepMind Control Suite and Atari Games showing 1.9x and 1.2x performance gains at the 100K environment and interaction steps benchmarks respectively. On the DeepMind Control Suite, CURL is the first image-based algorithm to nearly match the sample-efficiency of methods that use state-based features. Our code is open-sourced and available at https://github.com/MishaLaskin/curl.

First two authors contributed equally, website: https://mishalaskin.github.io/curl code: https://github.com/MishaLaskin/curl

References in corpus (22)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Soft Actor-Critic Algorithms and Applications
• Dueling Network Architectures for Deep Reinforcement Learning
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Noisy Networks for Exploration
• Contrastive Multiview Coding
• Model-Based Reinforcement Learning for Atari
• Learning to Navigate in Complex Environments
• Deep Reinforcement Learning that Matters
• Learning Latent Dynamics for Planning from Pixels
• On Mutual Information Maximization for Representation Learning
• Reinforcement Learning with Augmented Data
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• When to use parametric models in reinforcement learning?
• Loss is its own Reward: Self-Supervision for Reinforcement Learning

Cited by in corpus (11)

• Domain Generalization: A Survey
• Multi-Label Contrastive Learning for Abstract Visual Reasoning
• Learning Deformable Object Manipulation from Expert Demonstrations
• A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robots
• Cross-Modal Contrastive Learning of Representations for Navigation using Lightweight, Low-Cost Millimeter Wave Radar for Adverse Environmental Conditions
• Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning
• UnShadowNet: Illumination Critic Guided Contrastive Learning For Shadow Removal
• All by Myself: Learning Individualized Competitive Behaviour with a Contrastive Reinforcement Learning optimization
• Counterfactual experience augmented off-policy reinforcement learning