Consistency Regularization Can Improve Robustness to Label Noise

Abstract

Consistency regularization is a commonly-used technique for semi-supervised and self-supervised learning. It is an auxiliary objective function that encourages the prediction of the network to be similar in the vicinity of the observed training samples. Hendrycks et al. (2020) have recently shown such regularization naturally brings test-time robustness to corrupted data and helps with calibration. This paper empirically studies the relevance of consistency regularization for training-time robustness to noisy labels. First, we make two interesting and useful observations regarding the consistency of networks trained with the standard cross entropy loss on noisy datasets which are: (i) networks trained on noisy data have lower consistency than those trained on clean data, and(ii) the consistency reduces more significantly around noisy-labelled training data points than correctly-labelled ones. Then, we show that a simple loss function that encourages consistency improves the robustness of the models to label noise on both synthetic (CIFAR-10, CIFAR-100) and real-world (WebVision) noise as well as different noise rates and types and achieves state-of-the-art results.

arXiv commentPresented at the ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning. arXiv admin note: text overlap with arXiv:2105.04522

References in corpus (11)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• WebVision Database: Visual Learning and Understanding from Web Data
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Normalized Loss Functions for Deep Learning with Noisy Labels
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Are we done with ImageNet?
• Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels
• Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels