MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

Abstract

Recent deep networks are capable of memorizing the entire data even when the labels are completely random. To overcome the overfitting on corrupted labels, we propose a novel technique of learning another neural network, called MentorNet, to supervise the training of the base deep networks, namely, StudentNet. During training, MentorNet provides a curriculum (sample weighting scheme) for StudentNet to focus on the sample the label of which is probably correct. Unlike the existing curriculum that is usually predefined by human experts, MentorNet learns a data-driven curriculum dynamically with StudentNet. Experimental results demonstrate that our approach can significantly improve the generalization performance of deep networks trained on corrupted training data. Notably, to the best of our knowledge, we achieve the best-published result on WebVision, a large benchmark containing 2.2 million images of real-world noisy labels. The code are at https://github.com/google/mentornet

References in corpus (5)

• Nearly unbiased variable selection under minimax concave penalty
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Training Convolutional Networks with Noisy Labels
• A Closer Look at Memorization in Deep Networks

Cited by in corpus (79)

• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Dimensionality-Driven Learning with Noisy Labels
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis
• Hard Sample Aware Noise Robust Learning for Histopathology Image Classification
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations
• Combating noisy labels by agreement: A joint training method with co-regularization
• Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks
• Co-Correcting: Noise-tolerant Medical Image Classification via mutual Label Correction
• SELF: Learning to Filter Noisy Labels with Self-Ensembling
• On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification
• Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
• Weakly Supervised Vessel Segmentation in X-ray Angiograms by Self-Paced Learning from Noisy Labels with Suggestive Annotation
• An Entropic Optimal Transport Loss for Learning Deep Neural Networks under Label Noise in Remote Sensing Images
• Learning from Noisy Labels for Entity-Centric Information Extraction
• Regularizing Neural Network Training via Identity-wise Discriminative Feature Suppression
• Trustable Co-label Learning from Multiple Noisy Annotators
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• MESA: Boost Ensemble Imbalanced Learning with MEta-SAmpler
• Learning with Instance-Dependent Label Noise: A Sample Sieve Approach
• Uncertainty Based Detection and Relabeling of Noisy Image Labels
• MetaLabelNet: Learning to Generate Soft-Labels from Noisy-Labels
• Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels
• A Deep Model for Partial Multi-Label Image Classification with Curriculum Based Disambiguation
• Robust Learning Under Label Noise With Iterative Noise-Filtering
• Variational Rectification Inference for Learning with Noisy Labels
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Narrowing the Gap: Improved Detector Training with Noisy Location Annotations
• Understanding Generalization in Deep Learning via Tensor Methods
• Weakly Supervised AUC Optimization: A Unified Partial AUC Approach
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Needles in Haystacks: On Classifying Tiny Objects in Large Images
• Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs
• On Universal Black-Box Domain Adaptation
• PNT-Edge: Towards Robust Edge Detection with Noisy Labels by Learning Pixel-level Noise Transitions
• Improving Generalization by Controlling Label-Noise Information in Neural Network Weights
• Learning from Noisy Anchors for One-stage Object Detection
• Pick-and-Learn: Automatic Quality Evaluation for Noisy-Labeled Image Segmentation
• AutoAssist: A Framework to Accelerate Training of Deep Neural Networks
• Theory of Curriculum Learning, with Convex Loss Functions
• Robust and On-the-fly Dataset Denoising for Image Classification
• Bayesian Statistics Guided Label Refurbishment Mechanism: Mitigating Label Noise in Medical Image Classification
• Collaborative Label Correction via Entropy Thresholding
• Weakly Supervised Learning with Side Information for Noisy Labeled Images
• Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets
• Deep Bilevel Learning
• A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels
• Deep Learning from Small Amount of Medical Data with Noisy Labels: A Meta-Learning Approach
• On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
• NLNL: Negative Learning for Noisy Labels
• A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels
• Joint Negative and Positive Learning for Noisy Labels
• SelectNet: Learning to Sample from the Wild for Imbalanced Data Training
• Deep k-NN for Noisy Labels
• Product Image Recognition with Guidance Learning and Noisy Supervision
• Negative-ResNet: Noisy Ambulatory Electrocardiogram Signal Classification Scheme
• Facial Emotion Recognition with Noisy Multi-task Annotations
• Improving Medical Annotation Quality to Decrease Labeling Burden Using Stratified Noisy Cross-Validation
• More Information Supervised Probabilistic Deep Face Embedding Learning
• Limited Gradient Descent: Learning With Noisy Labels
• Informative Sample-Aware Proxy for Deep Metric Learning
• Learning from Noisy Similar and Dissimilar Data
• Combating Label Noise With A General Surrogate Model For Sample Selection
• Robust Collaborative Learning with Noisy Labels
• Alternating Loss Correction for Preterm-Birth Prediction from EHR Data with Noisy Labels
• HuCurl: Human-induced Curriculum Discovery
• Decoupled Gradient Harmonized Detector for Partial Annotation: Application to Signet Ring Cell Detection
• Mitigating Class Boundary Label Uncertainty to Reduce Both Model Bias and Variance
• Boosting Mapping Functionality of Neural Networks via Latent Feature Generation based on Reversible Learning
• Characterization of Posidonia Oceanica Seagrass Aerenchyma through Whole Slide Imaging: A Pilot Study
• Salvage Reusable Samples from Noisy Data for Robust Learning
• Learning advisor networks for noisy image classification
• Training Efficiency and Robustness in Deep Learning
• Suppressing Mislabeled Data via Grouping and Self-Attention
• Biquality Learning: a Framework to Design Algorithms Dealing with Closed-Set Distribution Shifts