Mitigating Memorization in Sample Selection for Learning with Noisy Labels

Abstract

Because deep learning is vulnerable to noisy labels, sample selection techniques, which train networks with only clean labeled data, have attracted a great attention. However, if the labels are dominantly corrupted by few classes, these noisy samples are called dominant-noisy-labeled samples, the network also learns dominant-noisy-labeled samples rapidly via content-aware optimization. In this study, we propose a compelling criteria to penalize dominant-noisy-labeled samples intensively through class-wise penalty labels. By averaging prediction confidences for the each observed label, we obtain suitable penalty labels that have high values if the labels are largely corrupted by some classes. Experiments were performed using benchmarks (CIFAR-10, CIFAR-100, Tiny-ImageNet) and real-world datasets (ANIMAL-10N, Clothing1M) to evaluate the proposed criteria in various scenarios with different noise rates. Using the proposed sample selection, the learning process of the network becomes significantly robust to noisy labels compared to existing methods in several noise types.

arXiv comment14 pages, 9 figures, spotlight presented at the ICML 2021 Workshop on Subset Selection in ML

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Temporal Ensembling for Semi-Supervised Learning
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Understanding deep learning requires rethinking generalization
• Classification with Noisy Labels by Importance Reweighting
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
• Unsupervised Label Noise Modeling and Loss Correction
• Masking: A New Perspective of Noisy Supervision
• Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
• Decoupling "when to update" from "how to update"
• Learning with Bad Training Data via Iterative Trimmed Loss Minimization
• Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
• Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels