Billion-scale semi-supervised learning for image classification

Abstract

This paper presents a study of semi-supervised learning with large convolutional networks. We propose a pipeline, based on a teacher/student paradigm, that leverages a large collection of unlabelled images (up to 1 billion). Our main goal is to improve the performance for a given target architecture, like ResNet-50 or ResNext. We provide an extensive analysis of the success factors of our approach, which leads us to formulate some recommendations to produce high-accuracy models for image classification with semi-supervised learning. As a result, our approach brings important gains to standard architectures for image, video and fine-grained classification. For instance, by leveraging one billion unlabelled images, our learned vanilla ResNet-50 achieves 81.2% top-1 accuracy on the ImageNet benchmark.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• YFCC100M: The New Data in Multimedia Research
• AutoAugment: Learning Augmentation Policies from Data
• A Closer Look at Spatiotemporal Convolutions for Action Recognition
• Rethinking ImageNet Pre-training
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification

Cited by in corpus (98)

• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• DINOv2: Learning Robust Visual Features without Supervision
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Rethinking Pre-training and Self-training
• An Overview of Deep Semi-Supervised Learning
• Self-training with Noisy Student improves ImageNet classification
• Compute and Energy Consumption Trends in Deep Learning Inference
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Toward Transformer-Based Object Detection
• SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Understanding and Improving Knowledge Distillation
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis
• Fixing the train-test resolution discrepancy: FixEfficientNet
• Partial success in closing the gap between human and machine vision
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• Self-training Improves Pre-training for Natural Language Understanding
• Self-learning for weakly supervised Gleason grading of local patterns
• Learning from Few Samples: A Survey
• Improving Semantic Segmentation via Self-Training
• SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
• ResNet strikes back: An improved training procedure in timm
• CyCNN: A Rotation Invariant CNN using Polar Mapping and Cylindrical Convolution Layers
• MiniVLM: A Smaller and Faster Vision-Language Model
• Scalable Transfer Learning with Expert Models
• WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition
• Teacher-Student chain for efficient semi-supervised histology image classification
• Supervised Transfer Learning at Scale for Medical Imaging
• Self-Training Vision Language BERTs with a Unified Conditional Model
• Fixing the train-test resolution discrepancy
• ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation
• Separating physically distinct mechanisms in complex infrared plasmonic nanostructures via machine learning enhanced electron energy loss spectroscopy
• Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
• A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning
• Omni-sourced Webly-supervised Learning for Video Recognition
• RDIS: Random Drop Imputation with Self-Training for Incomplete Time Series Data
• Semi-supervised learning using teacher-student models for vocal melody extraction
• Accelerating Sparse Deep Neural Networks
• ContextMix: A context-aware data augmentation method for industrial visual inspection systems
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Time for a Background Check! Uncovering the impact of Background Features on Deep Neural Networks
• More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval
• AutoGeoLabel: Automated Label Generation for Geospatial Machine Learning
• Webly Supervised Image Classification with Metadata: Automatic Noisy Label Correction via Visual-Semantic Graph
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• Modularity in Deep Learning: A Survey
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• Proof of Learning (PoLe): Empowering Machine Learning with Consensus Building on Blockchains
• A Novel Framework for Online Supervised Learning with Feature Selection
• Local Label Propagation for Large-Scale Semi-Supervised Learning
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
• ClusterFit: Improving Generalization of Visual Representations
• Lidar-Camera Co-Training for Semi-Supervised Road Detection
• Global Meets Local: Effective Multi-Label Image Classification via Category-Aware Weak Supervision
• Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
• Multi-Task Self-Training for Learning General Representations
• Towards Train-Test Consistency for Semi-supervised Temporal Action Localization
• Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
• Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data
• Skin disease diagnosis with deep learning: a review
• Self-Training for End-to-End Speech Translation
• Pseudo-labeling for Scalable 3D Object Detection
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
• Global Multiclass Classification and Dataset Construction via Heterogeneous Local Experts
• Classification of Melanocytic Nevus Images using BigTransfer (BiT)
• Progressive Representative Labeling for Deep Semi-Supervised Learning
• Representation Consolidation for Training Expert Students
• FooDI-ML: a large multi-language dataset of food, drinks and groceries images and descriptions
• Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts
• Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples
• Streaming Self-Training via Domain-Agnostic Unlabeled Images
• SODA: Detecting Covid-19 in Chest X-rays with Semi-supervised Open Set Domain Adaptation
• COBE: Contextualized Object Embeddings from Narrated Instructional Video
• Semi-Supervised Learning for In-Game Expert-Level Music-to-Dance Translation
• Webly Supervised Image Classification with Self-Contained Confidence
• Discover the Mysteries of the Maya: Selected Contributions from the Machine Learning Challenge & The Discovery Challenge Workshop at ECML PKDD 2021
• Position Paper: Towards Transparent Machine Learning
• ImageNet Pre-training also Transfers Non-Robustness
• Multimodal Knowledge Expansion
• Out-distribution aware Self-training in an Open World Setting
• Vehicle Trajectory Prediction by Transfer Learning of Semi-Supervised Models
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• Semi-supervised Learning with a Teacher-student Network for Generalized Attribute Prediction
• Improving Object Detection with Selective Self-supervised Self-training
• Tackling the Problem of Limited Data and Annotations in Semantic Segmentation
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Semi-Supervised Learning with Taxonomic Labels
• MapReader: A Computer Vision Pipeline for the Semantic Exploration of Maps at Scale
• Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation
• Relating CNNs with brain: Challenges and findings
• GAIA: A Transfer Learning System of Object Detection that Fits Your Needs