Learning Not to Learn in the Presence of Noisy Labels

Abstract

Learning in the presence of label noise is a challenging yet important task: it is crucial to design models that are robust in the presence of mislabeled datasets. In this paper, we discover that a new class of loss functions called the gambler's loss provides strong robustness to label noise across various levels of corruption. We show that training with this loss function encourages the model to "abstain" from learning on the data points with noisy labels, resulting in a simple and effective method to improve robustness and generalization. In addition, we propose two practical extensions of the method: 1) an analytical early stopping criterion to approximately stop training before the memorization of noisy labels, as well as 2) a heuristic for setting hyperparameters which do not require knowledge of the noise corruption rate. We demonstrate the effectiveness of our method by achieving strong results across three image and text classification tasks as compared to existing baselines.

References in corpus (8)

• Understanding deep learning requires rethinking generalization
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• SGD on Neural Networks Learns Functions of Increasing Complexity
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• L_DMI: An Information-theoretic Noise-robust Loss Function
• Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• LaProp: Separating Momentum and Adaptivity in Adam

Cited by in corpus (10)

• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Distributional Generalization: A New Kind of Generalization
• An Investigation of how Label Smoothing Affects Generalization
• Learning to Combat Noisy Labels via Classification Margins
• Model-Agnostic Meta-Learning for EEG Motor Imagery Decoding in Brain-Computer-Interfacing
• Differentiable Learning Under Triage
• Classification Under Human Assistance
• A Survey on Deep Learning with Noisy Labels: How to train your model when you cannot trust on the annotations?
• Volumization as a Natural Generalization of Weight Decay
• Regression Under Human Assistance