Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach

Abstract

We present a theoretically grounded approach to train deep neural networks, including recurrent networks, subject to class-dependent label noise. We propose two procedures for loss correction that are agnostic to both application domain and network architecture. They simply amount to at most a matrix inversion and multiplication, provided that we know the probability of each class being corrupted into another. We further show how one can estimate these probabilities, adapting a recent technique for noise estimation to the multi-class setting, and thus providing an end-to-end framework. Extensive experiments on MNIST, IMDB, CIFAR-10, CIFAR-100 and a large scale dataset of clothing images employing a diversity of architectures --- stacking dense, convolutional, pooling, dropout, batch normalization, word embedding, LSTM and residual layers --- demonstrate the noise robustness of our proposals. Incidentally, we also prove that, when ReLU is the only non-linearity, the loss curvature is immune to class-dependent label noise.

Oral paper at CVPR 2017

References in corpus (2)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Training Convolutional Networks with Noisy Labels

Cited by in corpus (6)

• Temporal Ensembling for Semi-Supervised Learning
• Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• On Universal Black-Box Domain Adaptation
• On the Robustness of Monte Carlo Dropout Trained with Noisy Labels