Sample Selection with Uncertainty of Losses for Learning with Noisy Labels

Abstract

In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled during training. However, losses are generated on-the-fly based on the model being trained with noisy labels, and thus large-loss data are likely but not certainly to be incorrect. There are actually two possibilities of a large-loss data point: (a) it is mislabeled, and then its loss decreases slower than other data, since deep neural networks "learn patterns first"; (b) it belongs to an underrepresented group of data and has not been selected yet. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the confidence intervals of losses derived from distribution-free concentration inequalities, but not losses themselves, are used for sample selection. In this way, we also give large-loss but less selected data a try; then, we can better distinguish between the cases (a) and (b) by seeing if the losses effectively decrease with the uncertainty after the try. As a result, we can better explore underrepresented data that are correctly labeled but seem to be mislabeled at first glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types.

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Empirical Evaluation of Rectified Activations in Convolutional Network
• General state space Markov chains and MCMC algorithms
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• Unsupervised Label Noise Modeling and Loss Correction
• Free Lunch for Few-shot Learning: Distribution Calibration
• Exponential and moment inequalities for U-statistics
• Normalized Loss Functions for Deep Learning with Noisy Labels
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Joint Optimization Framework for Learning with Noisy Labels
• Recent Advances in Algorithmic High-Dimensional Robust Statistics
• Robustness of Conditional GANs to Noisy Labels
• SIGUA: Forgetting May Make Learning with Noisy Labels More Robust
• Long-tail learning via logit adjustment
• Error-Bounded Correction of Noisy Labels
• Provably End-to-end Label-Noise Learning without Anchor Points
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels
• Class2Simi: A Noise Reduction Perspective on Learning with Noisy Labels
• Outlier Robust Mean Estimation with Subgaussian Rates via Stability
• Augmentation Strategies for Learning with Noisy Labels
• Coresets for Robust Training of Neural Networks against Noisy Labels
• What You See May Not Be What You Get: UCB Bandit Algorithms Robust to ε-Contamination