Robust Loss Functions under Label Noise for Deep Neural Networks

Abstract

In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.

arXiv commentAppeared in AAAI 2017

References in corpus (1)

• Training Convolutional Networks with Noisy Labels

Cited by in corpus (28)

• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• An Entropic Optimal Transport Loss for Learning Deep Neural Networks under Label Noise in Remote Sensing Images
• Decontamination of Mutual Contamination Models
• Joint Noise-Tolerant Learning and Meta Camera Shift Adaptation for Unsupervised Person Re-Identification
• ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks
• Machine Learning-Assisted Thermoelectric Cooling for On-Demand Multi-Hotspot Thermal Management
• QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extraction
• Webly Supervised Image Classification with Metadata: Automatic Noisy Label Correction via Visual-Semantic Graph
• Improving Regression Performance with Distributional Losses
• Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation
• Contrastive Learning Improves Model Robustness Under Label Noise
• From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation
• Combining Self-Supervised and Supervised Learning with Noisy Labels
• Weakly Supervised Learning with Side Information for Noisy Labeled Images
• Taming the Cross Entropy Loss
• The Resistance to Label Noise in K-NN and DNN Depends on its Concentration
• MFIF-GAN: A New Generative Adversarial Network for Multi-Focus Image Fusion
• On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
• Learning to Combat Noisy Labels via Classification Margins
• Detecting and Tracking Communal Bird Roosts in Weather Radar Data
• Towards Robust Pattern Recognition: A Review
• Ensemble Learning with Manifold-Based Data Splitting for Noisy Label Correction
• Joint Text and Label Generation for Spoken Language Understanding
• Do We Really Need Gold Samples for Sample Weighting Under Label Noise?
• Mixing between the Cross Entropy and the Expectation Loss Terms
• Skeptical Deep Learning with Distribution Correction