Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations

Abstract

We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training. We propose a semi-supervised loss, SuNCEt, based on noise-contrastive estimation and neighbourhood component analysis, that aims to distinguish examples of different classes in addition to the self-supervised instance-wise pretext tasks. On ImageNet, we find that SuNCEt can be used to match the semi-supervised learning accuracy of previous contrastive approaches while using less than half the amount of pre-training and compute. Our main insight is that leveraging even a small amount of labeled data during pre-training, and not only during fine-tuning, provides an important signal that can significantly accelerate contrastive learning of visual representations. Our code is available online at github.com/facebookresearch/suncet.

References in corpus (17)

• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Prototypical Networks for Few-shot Learning
• Bootstrap your own latent: A new approach to self-supervised Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Prototypical Contrastive Learning of Unsupervised Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks