A Simple Probabilistic Method for Deep Classification under Input-Dependent Label Noise

Abstract

Datasets with noisy labels are a common occurrence in practical applications of classification methods. We propose a simple probabilistic method for training deep classifiers under input-dependent (heteroscedastic) label noise. We assume an underlying heteroscedastic generative process for noisy labels. To make gradient based training feasible we use a temperature parameterized softmax as a smooth approximation to the assumed generative process. We illustrate that the softmax temperature controls a bias-variance trade-off for the approximation. By tuning the softmax temperature, we improve accuracy, log-likelihood and calibration on both image classification benchmarks with controlled label noise as well as Imagenet-21k which has naturally occurring label noise. For image segmentation, our method increases the mean IoU on the PASCAL VOC and Cityscapes datasets by more than 1% over the state-of-the-art model.

References in corpus (18)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Categorical Reparameterization with Gumbel-Softmax
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• WebVision Database: Visual Learning and Understanding from Web Data
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Single-Model Uncertainties for Deep Learning
• Are we done with ImageNet?
• Accurate Uncertainty Estimation and Decomposition in Ensemble Learning
• Mind the Nuisance: Gaussian Process Classification using Privileged Noise

Cited by in corpus (3)

• A Consensual Collaborative Learning Method for Remote Sensing Image Classification Under Noisy Multi-Labels
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model
• Deep Classifiers with Label Noise Modeling and Distance Awareness