Debiased Contrastive Learning

Abstract

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks. Theoretically, we establish generalization bounds for the downstream classification task.

References in corpus (14)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Prototypical Networks for Few-shot Learning
• Improved Baselines with Momentum Contrastive Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• Contrastive Multiview Coding
• DeepMind Control Suite
• An efficient framework for learning sentence representations
• Fairness Without Demographics in Repeated Loss Minimization
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Estimating the class prior and posterior from noisy positives and unlabeled data

Cited by in corpus (8)

• Hard Negative Mixing for Contrastive Learning
• Contrastive Learning Inverts the Data Generating Process
• Improving Text-to-Image Synthesis Using Contrastive Learning
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• A Unified Mixture-View Framework for Unsupervised Representation Learning
• Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning
• On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?