Billion-scale semi-supervised learning for image classification

Abstract

This paper presents a study of semi-supervised learning with large convolutional networks. We propose a pipeline, based on a teacher/student paradigm, that leverages a large collection of unlabelled images (up to 1 billion). Our main goal is to improve the performance for a given target architecture, like ResNet-50 or ResNext. We provide an extensive analysis of the success factors of our approach, which leads us to formulate some recommendations to produce high-accuracy models for image classification with semi-supervised learning. As a result, our approach brings important gains to standard architectures for image, video and fine-grained classification. For instance, by leveraging one billion unlabelled images, our learned vanilla ResNet-50 achieves 81.2% top-1 accuracy on the ImageNet benchmark.

References in corpus (2)

• Distilling the Knowledge in a Neural Network
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification

Cited by in corpus (37)

• An Overview of Deep Semi-Supervised Learning
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Toward Transformer-Based Object Detection
• SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Self-training Improves Pre-training for Natural Language Understanding
• Learning from Few Samples: A Survey
• Improving Semantic Segmentation via Self-Training
• CyCNN: A Rotation Invariant CNN using Polar Mapping and Cylindrical Convolution Layers
• Scalable Transfer Learning with Expert Models
• WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition
• Supervised Transfer Learning at Scale for Medical Imaging
• A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning
• Semi-supervised learning using teacher-student models for vocal melody extraction
• Accelerating Sparse Deep Neural Networks
• Time for a Background Check! Uncovering the impact of Background Features on Deep Neural Networks
• More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• Proof of Learning (PoLe): Empowering Machine Learning with Consensus Building on Blockchains
• Local Label Propagation for Large-Scale Semi-Supervised Learning
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
• ClusterFit: Improving Generalization of Visual Representations
• Lidar-Camera Co-Training for Semi-Supervised Road Detection
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
• Pseudo-labeling for Scalable 3D Object Detection
• Skin disease diagnosis with deep learning: a review
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
• Streaming Self-Training via Domain-Agnostic Unlabeled Images
• SODA: Detecting Covid-19 in Chest X-rays with Semi-supervised Open Set Domain Adaptation
• Semi-Supervised Learning for In-Game Expert-Level Music-to-Dance Translation
• Improving Object Detection with Selective Self-supervised Self-training
• Position Paper: Towards Transparent Machine Learning
• Webly Supervised Image Classification with Self-Contained Confidence
• Semi-supervised Learning with a Teacher-student Network for Generalized Attribute Prediction
• Out-distribution aware Self-training in an Open World Setting
• Tackling the Problem of Limited Data and Annotations in Semantic Segmentation
• FROST: Faster and more Robust One-shot Semi-supervised Training