Robust and On-the-fly Dataset Denoising for Image Classification
arXiv:2003.10647
Abstract
Memorization in over-parameterized neural networks could severely hurt generalization in the presence of mislabeled examples. However, mislabeled examples are hard to avoid in extremely large datasets collected with weak supervision. We address this problem by reasoning counterfactually about the loss distribution of examples with uniform random labels had they were trained with the real examples, and use this information to remove noisy examples from the training set. First, we observe that examples with uniform random labels have higher losses when trained with stochastic gradient descent under large learning rates. Then, we propose to model the loss distribution of the counterfactual examples using only the network parameters, which is able to model such examples with remarkable success. Finally, we propose to remove examples whose loss exceeds a certain quantile of the modeled loss distribution. This leads to On-the-fly Data Denoising (ODD), a simple yet effective algorithm that is robust to mislabeled examples, while introducing almost zero computational overhead compared to standard training. ODD is able to achieve state-of-the-art results on a wide range of datasets including real-world ones such as WebVision and Clothing1M.
References in corpus (24)
- mixup: Beyond Empirical Risk Minimization
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Wide Residual Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Understanding deep learning requires rethinking generalization
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
- Learning to Reweight Examples for Robust Deep Learning
- Classification with Noisy Labels by Importance Reweighting
- Training Deep Neural Networks on Noisy Labels with Bootstrapping
- MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
- Training Convolutional Networks with Noisy Labels
- Deep Learning is Robust to Massive Label Noise
- A Closer Look at Memorization in Deep Networks
- WebVision Database: Visual Learning and Understanding from Web Data
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
- Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
- Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
- Joint Optimization Framework for Learning with Noisy Labels
- Implicit Regularization in Deep Learning
- Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels
- An Alternative View: When Does SGD Escape Local Minima?
- L_DMI: An Information-theoretic Noise-robust Loss Function
- Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
- A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels