How does Disagreement Help Generalization against Label Corruption?

Abstract

Learning with noisy labels is one of the hottest problems in weakly-supervised learning. Based on memorization effects of deep neural networks, training on small-loss instances becomes very promising for handling noisy labels. This fosters the state-of-the-art approach "Co-teaching" that cross-trains two deep neural networks using the small-loss trick. However, with the increase of epochs, two networks converge to a consensus and Co-teaching reduces to the self-training MentorNet. To tackle this issue, we propose a robust learning paradigm called Co-teaching+, which bridges the "Update by Disagreement" strategy with the original Co-teaching. First, two networks feed forward and predict all data, but keep prediction disagreement data only. Then, among such disagreement data, each network selects its small-loss data, but back propagates the small-loss data from its peer network and updates its own parameters. Empirical results on benchmark datasets demonstrate that Co-teaching+ is much superior to many state-of-the-art methods in the robustness of trained models.

References in corpus (8)

• Classification with Noisy Labels by Importance Reweighting
• A Closer Look at Memorization in Deep Networks
• Joint Optimization Framework for Learning with Noisy Labels
• Robust Inference via Generative Classifiers for Handling Noisy Labels
• Learning from Noisy Labels with Distillation
• Learning From Noisy Large-Scale Datasets With Minimal Supervision
• Iterative Learning with Open-set Noisy Labels
• Theoretical Foundation of Co-Training and Disagreement-Based Algorithms

Cited by in corpus (53)

• Annotation-efficient deep learning for automatic medical image segmentation
• Denoising Implicit Feedback for Recommendation
• Early-Learning Regularization Prevents Memorization of Noisy Labels
• LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environment
• Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning
• A Survey of Label-noise Representation Learning: Past, Present and Future
• Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
• Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic
• Provably Consistent Partial-Label Learning
• Identifying Mislabeled Data using the Area Under the Margin Ranking
• Enhancing Diversity in Teacher-Student Networks via Asymmetric branches for Unsupervised Person Re-identification
• Label Noise Types and Their Effects on Deep Learning
• FINE Samples for Learning with Noisy Labels
• Training Binary Neural Networks through Learning with Noisy Supervision
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Learning with Instance-Dependent Label Noise: A Sample Sieve Approach
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels
• Understanding the Interaction of Adversarial Training with Noisy Labels
• Towards Noise-resistant Object Detection with Noisy Annotations
• ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks
• Rethinking Importance Weighting for Deep Learning under Distribution Shift
• A Second-Order Approach to Learning with Instance-Dependent Label Noise
• SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning
• Learning Frequency Domain Approximation for Binary Neural Networks
• Augmentation Strategies for Learning with Noisy Labels
• Co-training for Deep Object Detection: Comparing Single-modal and Multi-modal Approaches
• On Universal Black-Box Domain Adaptation
• Confidence Adaptive Regularization for Deep Learning with Noisy Labels
• Boosting Co-teaching with Compression Regularization for Label Noise
• Combining Self-Supervised and Supervised Learning with Noisy Labels
• Improving Generalization by Controlling Label-Noise Information in Neural Network Weights
• Multiple Fusion Adaptation: A Strong Framework for Unsupervised Semantic Segmentation Adaptation
• Noisy Annotation Refinement for Object Detection
• Learning to Combat Noisy Labels via Classification Margins
• On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
• Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond
• Joint Negative and Positive Learning for Noisy Labels
• Facial Emotion Recognition with Noisy Multi-task Annotations
• Distilling effective supervision for robust medical image segmentation with noisy labels
• Cross-denoising Network against Corrupted Labels in Medical Image Segmentation with Domain Shift
• More Information Supervised Probabilistic Deep Face Embedding Learning
• Transform consistency for learning with noisy labels
• P-DIFF: Learning Classifier with Noisy Labels based on Probability Difference Distributions
• A Survey on Deep Learning with Noisy Labels: How to train your model when you cannot trust on the annotations?
• CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning
• Boosting Semi-Supervised Face Recognition with Noise Robustness
• Sample Prior Guided Robust Model Learning to Suppress Noisy Labels
• Rethinking Noisy Label Models: Labeler-Dependent Noise with Adversarial Awareness
• Do We Really Need Gold Samples for Sample Weighting Under Label Noise?
• Using Under-trained Deep Ensembles to Learn Under Extreme Label Noise
• SENT: Sentence-level Distant Relation Extraction via Negative Training
• Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression
• Joint Text and Label Generation for Spoken Language Understanding