Self-training with Noisy Student improves ImageNet classification

Abstract

We present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant. Noisy Student Training achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2. Noisy Student Training extends the idea of self-training and distillation with the use of equal-or-larger student models and noise added to the student during learning. On ImageNet, we first train an EfficientNet model on labeled images and use it as a teacher to generate pseudo labels for 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the learning of the student, we inject noise such as dropout, stochastic depth, and data augmentation via RandAugment to the student so that the student generalizes better than the teacher. Models are available at https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet. Code is available at https://github.com/google-research/noisystudent.

arXiv commentCVPR 2020

References in corpus (37)

• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Unsupervised Data Augmentation for Consistency Training
• YFCC100M: The New Data in Multimedia Research
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Semi-Supervised Learning with Deep Generative Models
• Do Deep Nets Really Need to be Deep?
• Interpolation Consistency Training for Semi-Supervised Learning
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Dual Learning for Machine Translation
• Semi-Supervised Learning with Ladder Networks
• Do ImageNet Classifiers Generalize to ImageNet?
• Some Improvements on Deep Convolutional Neural Network Based Image Classification
• Learning with Pseudo-Ensembles
• Billion-scale semi-supervised learning for image classification
• Revisiting Semi-Supervised Learning with Graph Embeddings
• Auxiliary Deep Generative Models
• Learning to Self-Train for Semi-Supervised Few-Shot Classification
• Bayesian Dark Knowledge
• Natural Adversarial Examples
• Are Labels Required for Improving Adversarial Robustness?
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
• Revisiting Self-Training for Neural Sequence Generation
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• Domain Adaptive Transfer Learning with Specialist Models
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Unlabeled Data Improves Adversarial Robustness
• First-order Adversarial Vulnerability of Neural Networks and Input Dimension
• Batch Normalization is a Cause of Adversarial Vulnerability
• Semi-Supervised QA with Generative Domain-Adaptive Nets
• Using Videos to Evaluate Image Model Robustness
• Fixing the train-test resolution discrepancy
• Robustness properties of Facebook's ResNeXt WSL models
• Bridging the domain gap in cross-lingual document classification
• Semi-Supervised Segmentation of Salt Bodies in Seismic Images using an Ensemble of Convolutional Neural Networks
• EDF: Ensemble, Distill, and Fuse for Easy Video Labeling

Cited by in corpus (68)

• Big Self-Supervised Models are Strong Semi-Supervised Learners
• A Simple Semi-Supervised Learning Framework for Object Detection
• Image Augmentations for GAN Training
• Supervised Contrastive Learning
• Self-Training: A Survey
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning
• Are we done with ImageNet?
• Reference-guided Pseudo-Label Generation for Medical Semantic Segmentation
• Entropy-based Logic Explanations of Neural Networks
• Fixing the train-test resolution discrepancy: FixEfficientNet
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• ResKD: Residual-Guided Knowledge Distillation
• A Survey of Deep Learning for Scientific Discovery
• Adversarial Examples Improve Image Recognition
• Accounting for Variance in Machine Learning Benchmarks
• Noisy Differentiable Architecture Search
• Improving Semantic Segmentation via Self-Training
• Scaling Wide Residual Networks for Panoptic Segmentation
• Scalable Transfer Learning with Expert Models
• Teacher-Student chain for efficient semi-supervised histology image classification
• StyleGAN2 Distillation for Feed-forward Image Manipulation
• Semi-supervised learning using teacher-student models for vocal melody extraction
• Neuromorphologicaly-preserving Volumetric data encoding using VQ-VAE
• Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning
• Federated Semi-Supervised Learning for COVID Region Segmentation in Chest CT using Multi-National Data from China, Italy, Japan
• Noisy Self-Knowledge Distillation for Text Summarization
• Self-supervised Neural Architecture Search
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• Instance Credibility Inference for Few-Shot Learning
• Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• Estimating Galactic Distances From Images Using Self-supervised Representation Learning
• CHAIN: Concept-harmonized Hierarchical Inference Interpretation of Deep Convolutional Neural Networks
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Soft Labeling Affects Out-of-Distribution Detection of Deep Neural Networks
• Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance
• Bayesian Statistics Guided Label Refurbishment Mechanism: Mitigating Label Noise in Medical Image Classification
• Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
• Semi-Supervised Noisy Student Pre-training on EfficientNet Architectures for Plant Pathology Classification
• Asymmetric Rejection Loss for Fairer Face Recognition
• Towards Train-Test Consistency for Semi-supervised Temporal Action Localization
• Neural Networks Trained on Natural Scenes Exhibit Gestalt Closure
• Cheaper Pre-training Lunch: An Efficient Paradigm for Object Detection
• Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
• Progressive Representative Labeling for Deep Semi-Supervised Learning
• Combining Self-Training and Self-Supervised Learning for Unsupervised Disfluency Detection
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• COBE: Contextualized Object Embeddings from Narrated Instructional Video
• Multilevel Knowledge Transfer for Cross-Domain Object Detection
• Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples
• Can We Achieve More with Less? Exploring Data Augmentation for Toxic Comment Classification
• Empirical Perspectives on One-Shot Semi-supervised Learning
• Do CNNs Encode Data Augmentations?
• Removing Undesirable Feature Contributions Using Out-of-Distribution Data
• Deep learning for detecting pulmonary tuberculosis via chest radiography: an international study across 10 countries
• Semi-supervised Learning with a Teacher-student Network for Generalized Attribute Prediction
• Improving Object Detection with Selective Self-supervised Self-training
• Learning a Discriminant Latent Space with Neural Discriminant Analysis
• A general framework for defining and optimizing robustness
• Credal Self-Supervised Learning
• Cascade Bagging for Accuracy Prediction with Few Training Samples
• Noisy-LSTM: Improving Temporal Awareness for Video Semantic Segmentation
• Robust Temporal Ensembling for Learning with Noisy Labels
• Semi-supervised Learning for Dense Object Detection in Retail Scenes
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Negative Pseudo Labeling using Class Proportion for Semantic Segmentation in Pathology