L_DMI: An Information-theoretic Noise-robust Loss Function

Abstract

Accurately annotating large scale dataset is notoriously expensive both in time and in money. Although acquiring low-quality-annotated dataset can be much cheaper, it often badly damages the performance of trained models when using such dataset without particular treatment. Various methods have been proposed for learning with noisy labels. However, most methods only handle limited kinds of noise patterns, require auxiliary information or steps (e.g. , knowing or estimating the noise transition matrix), or lack theoretical justification. In this paper, we propose a novel information-theoretic loss function, ${\mathcal{L}}_{DMI}$, for training deep neural networks robust to label noise. The core of ${\mathcal{L}}_{DMI}$ is a generalized version of mutual information, termed Determinant based Mutual Information (DMI), which is not only information-monotone but also relatively invariant. \emph{To the best of our knowledge, ${\mathcal{L}}_{DMI}$ is the first loss function that is provably robust to instance-independent label noise, regardless of noise pattern, and it can be applied to any existing classification neural networks straightforwardly without any auxiliary information}. In addition to theoretical justification, we also empirically show that using ${\mathcal{L}}_{DMI}$ outperforms all other counterparts in the classification task on both image dataset and natural language dataset include Fashion-MNIST, CIFAR-10, Dogs vs. Cats, MR with a variety of synthesized noise patterns and noise amounts, as well as a real-world dataset Clothing1M. Codes are available at https://github.com/Newbeeer/L_DMI .

arXiv commentAccepted by NeurIPS 2019

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• mixup: Beyond Empirical Risk Minimization
• Learning deep representations by mutual information estimation and maximization
• Learning to Reweight Examples for Robust Deep Learning
• Classification with Noisy Labels by Importance Reweighting
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Training Convolutional Networks with Noisy Labels
• Noise Tolerance under Risk Minimization
• Dimensionality-Driven Learning with Noisy Labels
• Masking: A New Perspective of Noisy Supervision
• Mixture Proportion Estimation via Kernel Embedding of Distributions
• Learning from Binary Labels with Instance-Dependent Corruption

Cited by in corpus (15)

• Hard Sample Aware Noise Robust Learning for Histopathology Image Classification
• Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates
• Identifying Mislabeled Data using the Area Under the Margin Ranking
• Robust Long-Tailed Learning under Label Noise
• Learning Not to Learn in the Presence of Noisy Labels
• Learning Adaptive Loss for Robust Learning with Noisy Labels
• Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion
• Contrastive Learning Improves Model Robustness Under Label Noise
• SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learning
• Robust and On-the-fly Dataset Denoising for Image Classification
• An Ensemble Noise-Robust K-fold Cross-Validation Selection Method for Noisy Labels
• Learning to Combat Noisy Labels via Classification Margins
• Program Synthesis Over Noisy Data with Guarantees
• Sample Prior Guided Robust Model Learning to Suppress Noisy Labels
• Do We Really Need Gold Samples for Sample Weighting Under Label Noise?