SIGUA: Forgetting May Make Learning with Noisy Labels More Robust

Abstract

Given data with noisy labels, over-parameterized deep networks can gradually memorize the data, and fit everything in the end. Although equipped with corrections for noisy labels, many learning methods in this area still suffer overfitting due to undesired memorization. In this paper, to relieve this issue, we propose stochastic integrated gradient underweighted ascent (SIGUA): in a mini-batch, we adopt gradient descent on good data as usual, and learning-rate-reduced gradient ascent on bad data; the proposal is a versatile approach where data goodness or badness is w.r.t. desired or undesired memorization given a base learning method. Technically, SIGUA pulls optimization back for generalization when their goals conflict with each other; philosophically, SIGUA shows forgetting undesired memorization can reinforce desired memorization. Experiments demonstrate that SIGUA successfully robustifies two typical base learning methods, so that their performance is often significantly improved.

arXiv commentICML 2020 final version

Cited by in corpus (22)

• Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning
• A Survey of Label-noise Representation Learning: Past, Present and Future
• Do We Need Zero Training Loss After Achieving Zero Training Error?
• Understanding and Improving Early Stopping for Learning with Noisy Labels
• Provably Consistent Partial-Label Learning
• Sample Selection with Uncertainty of Losses for Learning with Noisy Labels
• Unsupervised Domain Adaptation of Black-Box Source Models
• Provably End-to-end Label-Noise Learning without Anchor Points
• Extended T: Learning with Mixed Closed-set and Open-set Noisy Labels
• Self-Damaging Contrastive Learning
• Rethinking Importance Weighting for Deep Learning under Distribution Shift
• A Second-Order Approach to Learning with Instance-Dependent Label Noise
• Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels
• Derivative Manipulation for General Example Weighting
• Instance Correction for Learning with Open-set Noisy Labels
• Deep Learning with Label Differential Privacy
• Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection
• Training Classifiers that are Universally Robust to All Label Noise Levels
• Binary Classification from Multiple Unlabeled Datasets via Surrogate Set Classification
• Constrained Instance and Class Reweighting for Robust Learning under Label Noise
• Combating Unknown Bias with Effective Bias-Conflicting Scoring and Gradient Alignment
• Estimating the electrical power output of industrial devices with end-to-end time-series classification in the presence of label noise