Curriculum Loss: Robust Learning and Generalization against Label Corruption

Abstract

Deep neural networks (DNNs) have great expressive power, which can even memorize samples with wrong labels. It is vitally important to reiterate robustness and generalization in DNNs against label corruption. To this end, this paper studies the 0-1 loss, which has a monotonic relationship with an empirical adversary (reweighted) risk~\citep{hu2016does}. Although the 0-1 loss has some robust properties, it is difficult to optimize. To efficiently optimize the 0-1 loss while keeping its robust properties, we propose a very simple and efficient loss, i.e. curriculum loss (CL). Our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses. Moreover, CL can adaptively select samples for model training. As a result, our loss can be deemed as a novel perspective of curriculum sample selection strategy, which bridges a connection between curriculum learning and robust learning. Experimental results on benchmark datasets validate the robustness of the proposed loss.

ICLR2020

References in corpus (8)

• Training Convolutional Networks with Noisy Labels
• A Closer Look at Memorization in Deep Networks
• Dimensionality-Driven Learning with Noisy Labels
• Masking: A New Perspective of Noisy Supervision
• Learning From Noisy Singly-labeled Data
• Does Distributionally Robust Supervised Learning Give Robust Classifiers?
• Marginalized Average Attentional Network for Weakly-Supervised Learning
• Robust Inference via Generative Classifiers for Handling Noisy Labels

Cited by in corpus (22)

• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• Robust Federated Learning with Noisy Labels
• Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic
• Open-set Label Noise Can Improve Robustness Against Inherent Label Noise
• Compositional Generalization by Learning Analytical Expressions
• Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularization
• Learning Adaptive Loss for Robust Learning with Noisy Labels
• Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation
• On Universal Black-Box Domain Adaptation
• Support Vector Machine Classifier via $L_{0/1}$ Soft-Margin Loss
• Combining Self-Supervised and Supervised Learning with Noisy Labels
• Adaptive Sample Selection for Robust Learning under Label Noise
• Bayesian Statistics Guided Label Refurbishment Mechanism: Mitigating Label Noise in Medical Image Classification
• On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
• Learning to Combat Noisy Labels via Classification Margins
• Robust binary classification with the 01 loss
• CCMN: A General Framework for Learning with Class-Conditional Multi-Label Noise
• Sample Prior Guided Robust Model Learning to Suppress Noisy Labels
• On the transferability of adversarial examples between convex and 01 loss models
• Towards adversarial robustness with 01 loss neural networks
• A Theoretical Analysis of Learning with Noisily Labeled Data
• Truth Discovery in Sequence Labels from Crowds