Normalized Loss Functions for Deep Learning with Noisy Labels

Abstract

Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss is not robust to noisy labels. Whilst new loss functions have been designed, they are only partially robust. In this paper, we theoretically show by applying a simple normalization that: any loss can be made robust to noisy labels. However, in practice, simply being robust is not sufficient for a loss function to train accurate DNNs. By investigating several robust loss functions, we find that they suffer from a problem of underfitting. To address this, we propose a framework to build robust loss functions called Active Passive Loss (APL). APL combines two robust loss functions that mutually boost each other. Experiments on benchmark datasets demonstrate that the family of new loss functions created by our APL framework can consistently outperform state-of-the-art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.

Accepted to ICML 2020

Cited by in corpus (31)

• Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression
• Understanding and Improving Early Stopping for Learning with Noisy Labels
• Sample Selection with Uncertainty of Losses for Learning with Noisy Labels
• Open-set Label Noise Can Improve Robustness Against Inherent Label Noise
• Extended T: Learning with Mixed Closed-set and Open-set Noisy Labels
• Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels
• One-Step Abductive Multi-Target Learning with Diverse Noisy Samples and Its Application to Tumour Segmentation for Breast Cancer
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels
• Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise
• SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning
• Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels
• Validation of the Practicability of Logical Assessment Formula for Evaluations with Inaccurate Ground-Truth Labels: An Application Study on Tumour Segmentation for Breast Cancer
• Confidence Adaptive Regularization for Deep Learning with Noisy Labels
• Instance Correction for Learning with Open-set Noisy Labels
• Deep Learning with Label Differential Privacy
• Robust Imitation Learning from Noisy Demonstrations
• Approximating Instance-Dependent Noise via Instance-Confidence Embedding
• Adaptive Sample Selection for Robust Learning under Label Noise
• Beyond Linear Subspace Clustering: A Comparative Study of Nonlinear Manifold Clustering Algorithms
• Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection
• Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond
• Joint Negative and Positive Learning for Noisy Labels
• Consistency Regularization Can Improve Robustness to Label Noise
• Training Classifiers that are Universally Robust to All Label Noise Levels
• Detecting Label Noise via Leave-One-Out Cross-Validation
• A Survey on Deep Learning with Noisy Labels: How to train your model when you cannot trust on the annotations?
• Using Early-Learning Regularization to Classify Real-World Noisy Data
• Constrained Instance and Class Reweighting for Robust Learning under Label Noise
• Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis
• A Symmetric Loss Perspective of Reliable Machine Learning
• An Exploration into why Output Regularization Mitigates Label Noise