Big Self-Supervised Models are Strong Semi-Supervised Learners

Abstract

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to common approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of big (deep and wide) networks during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9% ImageNet top-1 accuracy with just 1% of the labels ($\le$13 labeled images per class) using ResNet-50, a $10\times$ improvement in label efficiency over the previous state-of-the-art. With 10% of labels, ResNet-50 trained with our method achieves 77.5% top-1 accuracy, outperforming standard supervised training with all of the labels.

arXiv commentNeurIPS'2020. Code and pretrained models at https://github.com/google-research/simclr

References in corpus (25)

• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Scaling Laws for Neural Language Models
• Temporal Ensembling for Semi-Supervised Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Interpolation Consistency Training for Semi-Supervised Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Prototypical Contrastive Learning of Unsupervised Representations
• Semi-supervised Sequence Learning
• Deep Learning Scaling is Predictable, Empirically
• Large Scale Adversarial Representation Learning
• Learning with Pseudo-Ensembles
• Billion-scale semi-supervised learning for image classification
• Self-training with Noisy Student improves ImageNet classification

Cited by in corpus (175)

• Learning Transferable Visual Models From Natural Language Supervision
• Bootstrap your own latent: A new approach to self-supervised Learning
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Florence: A New Foundation Model for Computer Vision
• Hard Negative Mixing for Contrastive Learning
• Self-supervised learning methods and applications in medical imaging analysis: A survey
• An Overview of Deep Semi-Supervised Learning
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Revisiting ResNets: Improved Training and Scaling Strategies
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• Improving robustness against common corruptions by covariate shift adaptation
• Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
• Twin Contrastive Learning for Online Clustering
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Hybrid Dynamic Contrast and Probability Distillation for Unsupervised Person Re-Id
• Vector-quantized Image Modeling with Improved VQGAN
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Efficient Self-supervised Vision Transformers for Representation Learning
• Self-Supervised Representation Learning for Astronomical Images
• SEED: Self-supervised Distillation For Visual Representation
• Robust Pre-Training by Adversarial Contrastive Learning
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis
• Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
• Differentially Private Learning Needs Better Features (or Much More Data)
• Intriguing Properties of Contrastive Losses
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Federated Contrastive Learning for Volumetric Medical Image Segmentation
• Quasi Anomalous Knowledge: Searching for new physics with embedded knowledge
• MixCo: Mix-up Contrastive Learning for Visual Representation
• MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models
• Graph-MLP: Node Classification without Message Passing in Graph
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• Self-learning for weakly supervised Gleason grading of local patterns
• Whitening for Self-Supervised Representation Learning
• The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Multi-Label Contrastive Learning for Abstract Visual Reasoning
• Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation
• P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
• Contrastive Learning of Musical Representations
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• Self-supervised Contrastive Learning of Multi-view Facial Expressions
• Do Different Tracking Tasks Require Different Appearance Models?
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Multi-Format Contrastive Learning of Audio Representations
• A Study of the Generalizability of Self-Supervised Representations
• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning
• Self-supervised Learning for Large-scale Item Recommendations
• SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning
• Contrastive Learning Inverts the Data Generating Process
• MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
• Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis
• Hybrid Discriminative-Generative Training via Contrastive Learning
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• BYOL works even without batch statistics
• Model-Contrastive Federated Learning
• A benchmark for computational analysis of animal behavior, using animal-borne tags
• Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition
• Enhanced spatio-temporal electric load forecasts using less data with active deep learning
• Pushing the Limits of Non-Autoregressive Speech Recognition
• Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation
• Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning
• Deep residential representations: Using unsupervised learning to unlock elevation data for geo-demographic prediction
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types
• Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
• A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms
• Self-Damaging Contrastive Learning
• Domain Invariant Representation Learning with Domain Density Transformations
• Contrastive Learning Improves Model Robustness Under Label Noise
• Spatially Consistent Representation Learning
• Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Multimodal Semi-Supervised Learning for 3D Objects
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• Improving Transformation Invariance in Contrastive Representation Learning
• Self-supervised Mean Teacher for Semi-supervised Chest X-ray Classification
• Adversarially Adaptive Normalization for Single Domain Generalization
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancers
• For self-supervised learning, Rationality implies generalization, provably
• A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation
• Estimating Galactic Distances From Images Using Self-supervised Representation Learning
• Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR
• Semi-Supervised Classification and Segmentation on High Resolution Aerial Images
• Positional Contrastive Learning for Volumetric Medical Image Segmentation
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• Multiple Fusion Adaptation: A Strong Framework for Unsupervised Semantic Segmentation Adaptation
• Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework
• REX: Revisiting Budgeted Training with an Improved Schedule
• Learning Object-Centric Video Models by Contrasting Sets
• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
• Unsupervised Pre-training for Person Re-identification
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Self-supervised Video Representation Learning by Context and Motion Decoupling
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• Center-wise Local Image Mixture For Contrastive Representation Learning
• Cross-Domain Sentiment Classification with In-Domain Contrastive Learning
• Training Neural Networks with Fixed Sparse Masks
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery
• Multi-Task Self-Supervised Pre-Training for Music Classification
• Self-supervised Representation Learning with Relative Predictive Coding
• Towards Fine-grained Visual Representations by Combining Contrastive Learning with Image Reconstruction and Attention-weighted Pooling
• Polyphone Disambiguation in Mandarin Chinese with Semi-Supervised Learning
• StackMix: A complementary Mix algorithm
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• Are Fewer Labels Possible for Few-shot Learning?
• Unleashing the Potential of Unsupervised Pre-Training with Intra-Identity Regularization for Person Re-Identification
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Mean Embeddings with Test-Time Data Augmentation for Ensembling of Representations
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• Neural Audio Fingerprint for High-specific Audio Retrieval based on Contrastive Learning
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
• Multi-source Few-shot Domain Adaptation
• Aggregative Self-Supervised Feature Learning from a Limited Sample
• Meta Clustering Learning for Large-scale Unsupervised Person Re-identification
• Self-Supervised Learning from Unlabeled Fundus Photographs Improves Segmentation of the Retina
• Masked Contrastive Learning for Anomaly Detection
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• Large-Scale Unsupervised Person Re-Identification with Contrastive Learning
• Pretrained Encoders are All You Need
• eProduct: A Million-Scale Visual Search Benchmark to Address Product Recognition Challenges
• Semi-Supervised Vision Transformers
• ExCon: Explanation-driven Supervised Contrastive Learning for Image Classification
• Malfustection: Obfuscated Malware Detection and Malware Classification with Data Shortage by Combining Semi-Supervised and Contrastive Learning
• Personalizing Pre-trained Models
• Self-supervised Learning for Semi-supervised Temporal Language Grounding
• Dynamic Convolution for 3D Point Cloud Instance Segmentation
• Low Budget Active Learning via Wasserstein Distance: An Integer Programming Approach
• Contrastive Learning and Self-Training for Unsupervised Domain Adaptation in Semantic Segmentation
• Video Contrastive Learning with Global Context
• Resource and data efficient self supervised learning
• On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
• Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning
• Enabling the Network to Surf the Internet
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Complementary Relation Contrastive Distillation
• Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection
• An Image-based Approach of Task-driven Driving Scene Categorization
• SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations
• LiDAM: Semi-Supervised Learning with Localized Domain Adaptation and Iterative Matching
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• Improve Unsupervised Pretraining for Few-label Transfer
• Stochastic Contrastive Learning
• Stabilizing Label Assignment for Speech Separation by Self-supervised Pre-training
• Pretext Tasks selection for multitask self-supervised speech representation learning
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Deep Bregman Divergence for Contrastive Learning of Visual Representations
• Towards a Sample Efficient Reinforcement Learning Pipeline for Vision Based Robotics
• UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection
• Contrastive Active Inference
• Explicit homography estimation improves contrastive self-supervised learning
• TAGLETS: A System for Automatic Semi-Supervised Learning with Auxiliary Data
• Pedestrian Detection by Exemplar-Guided Contrastive Learning
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets
• Self-Supervised Learning for Large-Scale Unsupervised Image Clustering
• MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning
• Semi-Supervised Text Classification via Self-Pretraining
• Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation
• Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning
• Inverse Problems Leveraging Pre-trained Contrastive Representations
• Meta-Learning to Improve Pre-Training
• CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations