Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks

Abstract

Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy performance in many tasks. However, recent works have pointed out that the outputs provided by these models are not well-calibrated, seriously limiting their use in critical decision scenarios. In this work, we propose to use a decoupled Bayesian stage, implemented with a Bayesian Neural Network (BNN), to map the uncalibrated probabilities provided by a DNN to calibrated ones, consistently improving calibration. Our results evidence that incorporating uncertainty provides more reliable probabilistic models, a critical condition for achieving good calibration. We report a generous collection of experimental results using high-accuracy DNNs in standardized image classification benchmarks, showing the good performance, flexibility and robust behavior of our approach with respect to several state-of-the-art calibration methods. Code for reproducibility is provided.

arXiv commentSubmit to Neurocomputing

References in corpus (30)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Explaining and Harnessing Adversarial Examples
• End to End Learning for Self-Driving Cars
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Squeeze-and-Excitation Networks
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Variational Inference with Normalizing Flows
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• A Conceptual Introduction to Hamiltonian Monte Carlo
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout and the Local Reparameterization Trick
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Auxiliary Deep Generative Models
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Amortized Inference Regularization
• Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification
• Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes
• Calibrating Deep Convolutional Gaussian Processes
• Ergodic Inference: Accelerate Convergence by Optimisation

Cited by in corpus (7)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Transforming Gaussian Processes With Normalizing Flows
• Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• On Calibration of Mixup Training for Deep Neural Networks
• Improving Calibration and Out-of-Distribution Detection in Medical Image Segmentation with Convolutional Neural Networks
• h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective