Intriguing Properties of Contrastive Losses

Abstract

We study three intriguing properties of contrastive learning. First, we generalize the standard contrastive loss to a broader family of losses, and we find that various instantiations of the generalized loss perform similarly under the presence of a multi-layer non-linear projection head. Second, we study if instance-based contrastive learning (with a global image representation) can learn well on images with multiple objects present. We find that meaningful hierarchical local features can be learned despite the fact that these objectives operate on global instance-level features. Finally, we study the phenomenon of feature suppression among competing features shared across augmented views, such as "color distribution" vs "object class". We construct datasets with explicit and controllable competing features, and show that, for contrastive learning, a few bits of easy-to-learn shared features can suppress, and even fully prevent, the learning of other sets of competing features. In scenarios where there are multiple objects in an image, the dominant object would suppress the learning of smaller objects. Existing contrastive learning methods critically rely on data augmentation to favor certain sets of features over others, and could suffer from learning saturation for scenarios where existing augmentations cannot fully address the feature suppression. This poses open challenges to existing contrastive learning techniques.

arXiv commentNeurIPS 2021. Code and visualization at https://contrastive-learning.github.io/intriguing

References in corpus (23)

• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• Deep Residual Learning for Image Recognition
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Prototypical Contrastive Learning of Unsupervised Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• On Mutual Information Maximization for Representation Learning
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• The Pitfalls of Simplicity Bias in Neural Networks
• What makes instance discrimination good for transfer learning?
• What shapes feature representations? Exploring datasets, architectures, and training
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Understanding Self-supervised Learning with Dual Deep Networks
• Understanding the Limitations of Variational Mutual Information Estimators
• Contrastive learning, multi-view redundancy, and linear models

Cited by in corpus (14)

• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Self-Supervised Learning with Kernel Dependence Maximization
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• Elsa: Energy-based learning for semi-supervised anomaly detection
• Denoising-While-Completing Network (DWCNet): Robust Point Cloud Completion Under Corruption
• Unsupervised domain adaptation via coarse-to-fine feature alignment method using contrastive learning
• Addressing Feature Suppression in Unsupervised Visual Representations
• Deep Bregman Divergence for Contrastive Learning of Visual Representations
• More Robust Dense Retrieval with Contrastive Dual Learning
• Improve Unsupervised Pretraining for Few-label Transfer
• Inverse Problems Leveraging Pre-trained Contrastive Representations