Robust Temporal Ensembling for Learning with Noisy Labels

Abstract

Successful training of deep neural networks with noisy labels is an essential capability as most real-world datasets contain some amount of mislabeled data. Left unmitigated, label noise can sharply degrade typical supervised learning approaches. In this paper, we present robust temporal ensembling (RTE), which combines robust loss with semi-supervised regularization methods to achieve noise-robust learning. We demonstrate that RTE achieves state-of-the-art performance across the CIFAR-10, CIFAR-100, ImageNet, WebVision, and Food-101N datasets, while forgoing the recent trend of label filtering and/or fixing. Finally, we show that RTE also retains competitive corruption robustness to unforeseen input noise using CIFAR-10-C, obtaining a mean corruption error (mCE) of 13.50% even in the presence of an 80% noise ratio, versus 26.9% mCE with standard methods on clean data.

arXiv commentincludes additional baselines and hyperparam references

References in corpus (16)

• Wide Residual Networks
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Learning to Reweight Examples for Robust Deep Learning
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• Deep Learning Scaling is Predictable, Empirically
• WebVision Database: Visual Learning and Understanding from Web Data
• Self-training with Noisy Student improves ImageNet classification
• Identity Mappings in Deep Residual Networks
• Google matrix analysis of directed networks
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
• Combating Label Noise in Deep Learning Using Abstention
• Robust Inference via Generative Classifiers for Handling Noisy Labels
• Learning from Noisy Labels with Distillation
• A Generalized Framework for Population Based Training