Label Refinery: Improving ImageNet Classification through Label Progression

Abstract

Among the three main components (data, labels, and models) of any supervised learning system, data and models have been the main subjects of active research. However, studying labels and their properties has received very little attention. Current principles and paradigms of labeling impose several challenges to machine learning algorithms. Labels are often incomplete, ambiguous, and redundant. In this paper we study the effects of various properties of labels and introduce the Label Refinery: an iterative procedure that updates the ground truth labels after examining the entire dataset. We show significant gain using refined labels across a wide range of models. Using a Label Refinery improves the state-of-the-art top-1 accuracy of (1) AlexNet from 59.3 to 67.2, (2) MobileNet from 70.6 to 73.39, (3) MobileNet-0.25 from 50.6 to 55.59, (4) VGG19 from 72.7 to 75.46, and (5) Darknet19 from 72.9 to 74.47.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• FitNets: Hints for Thin Deep Nets
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Verb Semantics and Lexical Selection
• Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentation
• Learning from Noisy Labels with Distillation
• A hierarchical loss and its problems when classifying non-hierarchically

Cited by in corpus (48)

• Knowledge Distillation: A Survey
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks
• Self-Adaptive Training: beyond Empirical Risk Minimization
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
• Relational Knowledge Distillation
• Physical Knowledge Enhanced Deep Neural Network for Sea Surface Temperature Prediction
• Synergic Adversarial Label Learning for Grading Retinal Diseases via Knowledge Distillation and Multi-task Learning
• Confidence Regularized Self-Training
• Unsupervised Domain Adaptation for RF-based Gesture Recognition
• Regularizing Class-wise Predictions via Self-knowledge Distillation
• Towards Understanding Label Smoothing
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Exploration of Class Center for Fine-Grained Visual Classification
• Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
• Adaptive Regularization of Labels
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision
• Multiple Expert Brainstorming for Domain Adaptive Person Re-identification
• Universally Slimmable Networks and Improved Training Techniques
• FQ-Conv: Fully Quantized Convolution for Efficient and Accurate Inference
• Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification
• MEAL: Multi-Model Ensemble via Adversarial Learning
• Efficient Deep Neural Networks
• Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval
• MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
• Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• ISP Distillation
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• Classes Matter: A Fine-grained Adversarial Approach to Cross-domain Semantic Segmentation
• Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
• Towards Practical Lipreading with Distilled and Efficient Models
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• MSVQ: Self-Supervised Learning with Multiple Sample Views and Queues
• Object Detection for Understanding Assembly Instruction Using Context-aware Data Augmentation and Cascade Mask R-CNN
• Improving Binary Neural Networks through Fully Utilizing Latent Weights
• Energy-constrained Self-training for Unsupervised Domain Adaptation
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation
• Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement
• Enhancing Segment-Based Speech Emotion Recognition by Deep Self-Learning
• A New Training Framework for Deep Neural Network
• GREEN: a Graph REsidual rE-ranking Network for Grading Diabetic Retinopathy
• Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation
• Label Augmentation via Time-based Knowledge Distillation for Financial Anomaly Detection
• Min-Entropy Latent Model for Weakly Supervised Object Detection
• Differential Replication in Machine Learning
• Knowledge Distillation with Noisy Labels for Natural Language Understanding