Confidence Calibration for Convolutional Neural Networks Using Structured Dropout

Abstract

In classification applications, we often want probabilistic predictions to reflect confidence or uncertainty. Dropout, a commonly used training technique, has recently been linked to Bayesian inference, yielding an efficient way to quantify uncertainty in neural network models. However, as previously demonstrated, confidence estimates computed with a naive implementation of dropout can be poorly calibrated, particularly when using convolutional networks. In this paper, through the lens of ensemble learning, we associate calibration error with the correlation between the models sampled with dropout. Motivated by this, we explore the use of structured dropout to promote model diversity and improve confidence calibration. We use the SVHN, CIFAR-10 and CIFAR-100 datasets to empirically compare model diversity and confidence errors obtained using various dropout techniques. We also show the merit of structured dropout in a Bayesian active learning application.

References in corpus (9)

• DropBlock: A regularization method for convolutional networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout and the Local Reparameterization Trick
• Shake-Shake regularization
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• The Description Length of Deep Learning Models
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
• Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation

Cited by in corpus (9)

• Ensembling Low Precision Models for Binary Biomedical Image Segmentation
• A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks
• DropCluster: A structured dropout for convolutional networks
• Intelligence plays dice: Stochasticity is essential for machine learning
• The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Learning to Cascade: Confidence Calibration for Improving the Accuracy and Computational Cost of Cascade Inference Systems
• Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One
• Learning from Matured Dumb Teacher for Fine Generalization