Supervised Contrastive Learning

Abstract

Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models. Modern batch contrastive approaches subsume or significantly outperform traditional contrastive losses such as triplet, max-margin and the N-pairs loss. In this work, we extend the self-supervised batch contrastive approach to the fully-supervised setting, allowing us to effectively leverage label information. Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes. We analyze two possible versions of the supervised contrastive (SupCon) loss, identifying the best-performing formulation of the loss. On ResNet-200, we achieve top-1 accuracy of 81.4% on the ImageNet dataset, which is 0.8% above the best number reported for this architecture. We show consistent outperformance over cross-entropy on other datasets and two ResNet variants. The loss shows benefits for robustness to natural corruptions and is more stable to hyperparameter settings such as optimizers and data augmentations. Our loss function is simple to implement, and reference TensorFlow code is released at https://t.ly/supcon.

References in corpus (21)

• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Explaining and Harnessing Adversarial Examples
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• An overview of gradient descent optimization algorithms
• mixup: Beyond Empirical Risk Minimization
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Training Convolutional Networks with Noisy Labels
• Large Batch Training of Convolutional Networks
• Hyperparameter Search in Machine Learning
• On Mutual Information Maximization for Representation Learning
• Self-training with Noisy Student improves ImageNet classification
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Massive Exploration of Neural Machine Translation Architectures
• Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
• Cross-Entropy Loss and Low-Rank Features Have Responsibility for Adversarial Examples

Cited by in corpus (69)

• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Self-supervised Co-training for Video Representation Learning
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Margin Preserving Self-paced Contrastive Learning Towards Domain Adaptation for Medical Image Segmentation
• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
• Contrastive Learning with Adversarial Examples
• Multi-Label Contrastive Learning for Abstract Visual Reasoning
• ResNet strikes back: An improved training procedure in timm
• Self-supervised Contrastive Learning of Multi-view Facial Expressions
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Self-Supervised Relational Reasoning for Representation Learning
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• Enriched Music Representations with Multiple Cross-modal Contrastive Learning
• Hybrid Discriminative-Generative Training via Contrastive Learning
• SoundCLR: Contrastive Learning of Representations For Improved Environmental Sound Classification
• On Episodes, Prototypical Networks, and Few-shot Learning
• Deep Partition Aggregation: Provable Defense against General Poisoning Attacks
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
• Category Contrast for Unsupervised Domain Adaptation in Visual Tasks
• Multimodal Fusion Refiner Networks
• Supporting Clustering with Contrastive Learning
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Adversarial Training with Contrastive Learning in NLP
• Contrastive Semi-Supervised Learning for 2D Medical Image Segmentation
• Mini-batch graphs for robust image classification
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models
• Contrastive Representation Learning for Whole Brain Cytoarchitectonic Mapping in Histological Human Brain Sections
• Contrastive Learning for Robust Android Malware Familial Classification
• CoKe: Localized Contrastive Learning for Robust Keypoint Detection
• The NPU System for the 2020 Personalized Voice Trigger Challenge
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• Does Data Augmentation Benefit from Split BatchNorms
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery
• Improving Contrastive Learning by Visualizing Feature Transformation
• SleepPriorCL: Contrastive Representation Learning with Prior Knowledge-based Positive Mining and Adaptive Temperature for Sleep Staging
• Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations
• Distribution Estimation to Automate Transformation Policies for Self-Supervision
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• StackMix: A complementary Mix algorithm
• Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning
• Robust Representation Learning via Perceptual Similarity Metrics
• Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data
• Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning
• Self-supervised Regularization for Text Classification
• ExCon: Explanation-driven Supervised Contrastive Learning for Image Classification
• MixCon: Adjusting the Separability of Data Representations for Harder Data Recovery
• Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain
• CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning
• COIN: Contrastive Identifier Network for Breast Mass Diagnosis in Mammography
• UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
• Multi-level Knowledge Distillation via Knowledge Alignment and Correlation
• WildGait: Learning Gait Representations from Raw Surveillance Streams
• Similarity Guided Deep Face Image Retrieval
• Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval
• UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection
• Towards noise robust trigger-word detection with contrastive learning pre-task for fast on-boarding of new trigger-words
• Efficient Classification of Very Large Images with Tiny Objects
• Video Class Agnostic Segmentation with Contrastive Learning for Autonomous Driving
• ConCAD: Contrastive Learning-based Cross Attention for Sleep Apnea Detection
• Graph Partner Neural Networks for Semi-Supervised Learning on Graphs
• Adaptive Hierarchical Similarity Metric Learning with Noisy Labels
• Exploiting Class Similarity for Machine Learning with Confidence Labels and Projective Loss Functions
• Dimension Free Generalization Bounds for Non Linear Metric Learning
• Class-Incremental Domain Adaptation with Smoothing and Calibration for Surgical Report Generation
• Contrast R-CNN for Continual Learning in Object Detection