Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels

Abstract

Prior works have found it beneficial to combine provably noise-robust loss functions e.g., mean absolute error (MAE) with standard categorical loss function e.g. cross entropy (CE) to improve their learnability. Here, we propose to use Jensen-Shannon divergence as a noise-robust loss function and show that it interestingly interpolate between CE and MAE with a controllable mixing parameter. Furthermore, we make a crucial observation that CE exhibit lower consistency around noisy data points. Based on this observation, we adopt a generalized version of the Jensen-Shannon divergence for multiple distributions to encourage consistency around data points. Using this loss function, we show state-of-the-art results on both synthetic (CIFAR), and real-world (e.g., WebVision) noise with varying noise rates.

arXiv commentNeural Information Processing Systems (NeurIPS 2021)

References in corpus (16)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• WebVision Database: Visual Learning and Understanding from Web Data
• On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Are Anchor Points Really Indispensable in Label-Noise Learning?
• Masking: A New Perspective of Noisy Supervision
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Normalized Loss Functions for Deep Learning with Noisy Labels
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Are we done with ImageNet?
• Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels
• Learning with Feature-Dependent Label Noise: A Progressive Approach
• Consistency Regularization for Adversarial Robustness

Cited by in corpus (4)

• Variational Rectification Inference for Learning with Noisy Labels
• Consistency Regularization Can Improve Robustness to Label Noise
• Does Adversarial Oversampling Help us?
• CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning