Conditional Negative Sampling for Contrastive Learning of Visual Representations

Abstract

Recent methods for learning unsupervised visual representations, dubbed contrastive learning, optimize the noise-contrastive estimation (NCE) bound on mutual information between two views of an image. NCE uses randomly sampled negative examples to normalize the objective. In this paper, we show that choosing difficult negatives, or those more similar to the current instance, can yield stronger representations. To do this, we introduce a family of mutual information estimators that sample negatives conditionally -- in a "ring" around each positive. We prove that these estimators lower-bound mutual information, with higher bias but lower variance than NCE. Experimentally, we find our approach, applied on top of existing models (IR, CMC, and MoCo) improves accuracy by 2-5% points in each case, measured by linear evaluation on four standard image datasets. Moreover, we find continued benefits when transferring features to a variety of new image distributions from the Meta-Dataset collection and to a variety of downstream tasks such as object detection, instance segmentation, and keypoint detection.

8 pages, 4 pages supplement

References in corpus (12)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Language Models are Few-Shot Learners
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• On Mutual Information Maximization for Representation Learning
• Self-labelling via simultaneous clustering and representation learning

Cited by in corpus (8)

• Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning
• Temperature as Uncertainty in Contrastive Learning
• DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning
• Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations