Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels

Abstract

Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called Co-teaching for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

NIPS 2018 camera-ready version

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Learning to Reweight Examples for Robust Deep Learning
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• A Closer Look at Memorization in Deep Networks
• Dimensionality-Driven Learning with Noisy Labels
• Positive-Unlabeled Learning with Non-Negative Risk Estimator
• Joint Optimization Framework for Learning with Noisy Labels
• Learning from Noisy Labels with Distillation
• Learning From Noisy Large-Scale Datasets With Minimal Supervision
• Iterative Learning with Open-set Noisy Labels
• On the Convergence of A Family of Robust Losses for Stochastic Gradient Descent

Cited by in corpus (26)

• Annotation-efficient deep learning for automatic medical image segmentation
• GhostNets on Heterogeneous Devices via Cheap Operations
• Active label cleaning for improved dataset quality under resource constraints
• Self-Training: A Survey
• Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic
• The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track
• A Deep Model for Partial Multi-Label Image Classification with Curriculum Based Disambiguation
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Co-training for Deep Object Detection: Comparing Single-modal and Multi-modal Approaches
• Learning with Biased Complementary Labels
• A Two-Stream Mutual Attention Network for Semi-supervised Biomedical Segmentation with Noisy Labels
• Task Agnostic Robust Learning on Corrupt Outputs by Correlation-Guided Mixture Density Networks
• Label-Noise Robust Generative Adversarial Networks
• NLNL: Negative Learning for Noisy Labels
• On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
• Safeguarded Dynamic Label Regression for Generalized Noisy Supervision
• Limited Gradient Descent: Learning With Noisy Labels
• Combating Label Noise With A General Surrogate Model For Sample Selection
• Instance-dependent Noisy-label Learning with Graphical Model Based Noise-rate Estimation
• Training Classifiers that are Universally Robust to All Label Noise Levels
• Noisy Label Classification using Label Noise Selection with Test-Time Augmentation Cross-Entropy and NoiseMix Learning
• Mitigating Memorization in Sample Selection for Learning with Noisy Labels
• Efficient Classification of Very Large Images with Tiny Objects
• CrowdTeacher: Robust Co-teaching with Noisy Answers & Sample-specific Perturbations for Tabular Data
• Learning advisor networks for noisy image classification