How does Early Stopping Help Generalization against Label Noise?

Abstract

Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overfitting to the noisy labels. In this paper, we claim that such overfitting can be avoided by "early stopping" training a deep neural network before the noisy labels are severely memorized. Then, we resume training the early stopped network using a "maximal safe set," which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point. Putting them all together, our novel two-phase training method, called Prestopping, realizes noise-free training under any type of label noise for practical use. Extensive experiments using four image benchmark data sets verify that our method significantly outperforms four state-of-the-art methods in test error by 0.4-8.2 percent points under existence of real-world noise.

arXiv commentInternational Conference on Machine Learning, Workshop on Uncertainty and Robustness in Deep Learning. See: https://sites.google.com/view/udlworkshop2020/home

References in corpus (8)

• Learning to Reweight Examples for Robust Deep Learning
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• A Closer Look at Memorization in Deep Networks
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Learning with Bad Training Data via Iterative Trimmed Loss Minimization
• Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels
• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian

Cited by in corpus (3)

• Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection
• Learning to Combat Noisy Labels via Classification Margins
• Carpe Diem, Seize the Samples Uncertain "At the Moment" for Adaptive Batch Selection