Limited Gradient Descent: Learning With Noisy Labels
arXiv:1811.08117
Abstract
Label noise may affect the generalization of classifiers, and the effective learning of main patterns from samples with noisy labels is an important challenge. Recent studies have shown that deep neural networks tend to prioritize the learning of simple patterns over the memorization of noise patterns. This suggests a possible method to search for the best generalization that learns the main pattern until the noise begins to be memorized. Traditional approaches often employ a clean validation set to find the best stop timing of learning, i.e., early stopping. However, the generalization performance of such methods relies on the quality of validation sets. Further, in practice, a clean validation set is sometimes difficult to obtain. To solve this problem, we propose a method that can estimate the optimal stopping timing without a clean validation set, called limited gradient descent. We modified the labels of a few samples in a noisy dataset to obtain false labels and to create a reverse pattern. By monitoring the learning progress of the noisy and reverse samples, we can determine the stop timing of learning. In this paper, we also theoretically provide some necessary conditions on learning with noisy labels. Experimental results on CIFAR-10 and CIFAR-100 datasets demonstrate that our approach has a comparable generalization performance to methods relying on a clean validation set. Thus, on the noisy Clothing-1M dataset, our approach surpasses methods that rely on a clean validation set.
References in corpus (16)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- mixup: Beyond Empirical Risk Minimization
- YFCC100M: The New Data in Multimedia Research
- Understanding deep learning requires rethinking generalization
- Training Deep Neural Networks on Noisy Labels with Bootstrapping
- MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
- Training Convolutional Networks with Noisy Labels
- EMNIST: an extension of MNIST to handwritten letters
- Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
- A Closer Look at Memorization in Deep Networks
- Learning Discrete Representations via Information Maximizing Self-Augmented Training
- Making Risk Minimization Tolerant to Label Noise
- Mixture Proportion Estimation via Kernel Embedding of Distributions
- Learning from Noisy Labels with Distillation
- Learning From Noisy Large-Scale Datasets With Minimal Supervision
- Auxiliary Image Regularization for Deep CNNs with Noisy Labels