Early-Learning Regularization Prevents Memorization of Noisy Labels

Abstract

We propose a novel framework to perform classification via deep learning in the presence of noisy annotations. When trained on noisy labels, deep neural networks have been observed to first fit the training data with clean labels during an "early learning" phase, before eventually memorizing the examples with false labels. We prove that early learning and memorization are fundamental phenomena in high-dimensional classification tasks, even in simple linear models, and give a theoretical explanation in this setting. Motivated by these findings, we develop a new technique for noisy classification tasks, which exploits the progress of the early learning phase. In contrast with existing approaches, which use the model output during early learning to detect the examples with clean labels, and either ignore or attempt to correct the false labels, we take a different route and instead capitalize on early learning via regularization. There are two key elements to our approach. First, we leverage semi-supervised learning techniques to produce target probabilities based on the model outputs. Second, we design a regularization term that steers the model towards these targets, implicitly preventing memorization of the false labels. The resulting framework is shown to provide robustness to noisy annotations on several standard benchmarks and real-world datasets, where it achieves results comparable to the state of the art.

References in corpus (13)

• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Learning to Reweight Examples for Robust Deep Learning
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• A Closer Look at Memorization in Deep Networks
• WebVision Database: Visual Learning and Understanding from Web Data
• Dimensionality-Driven Learning with Noisy Labels
• Unsupervised Label Noise Modeling and Loss Correction
• Are Anchor Points Really Indispensable in Label-Noise Learning?
• How does Disagreement Help Generalization against Label Corruption?
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters
• Learning without Concentration

Cited by in corpus (12)

• Active label cleaning for improved dataset quality under resource constraints
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification
• Cross-head Supervision for Crowd Counting with Noisy Annotations
• Local Label Point Correction for Edge Detection of Overlapping Cervical Cells
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels
• Variational Rectification Inference for Learning with Noisy Labels
• DyGen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling
• Unreliable Partial Label Learning with Recursive Separation
• Distilling effective supervision for robust medical image segmentation with noisy labels
• Combating Label Noise With A General Surrogate Model For Sample Selection
• Sample Prior Guided Robust Model Learning to Suppress Noisy Labels