Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation

Abstract

Many methods have been proposed to quantify the predictive uncertainty associated with the outputs of deep neural networks. Among them, ensemble methods often lead to state-of-the-art results, though they require modifications to the training procedures and are computationally costly for both training and inference. In this paper, we propose a new single-model based approach. The main idea is inspired by the observation that we can "simulate" an ensemble of models by drawing from a Gaussian distribution, with a form similar to those from the asymptotic normality theory, infinitesimal Jackknife, Laplacian approximation to Bayesian neural networks, and trajectories in stochastic gradient descents. However, instead of using each model in the "ensemble" to predict and then aggregating their predictions, we integrate the Gaussian distribution and the softmax outputs of the neural networks. We use a mean-field approximation formula to compute this analytically intractable integral. The proposed approach has several appealing properties: it functions as an ensemble without requiring multiple models, and it enables closed-form approximate inference using only the first and second moments of the Gaussian. Empirically, the proposed approach performs competitively when compared to state-of-the-art methods, including deep ensembles, temperature scaling, dropout and Bayesian NNs, on standard uncertainty estimation tasks. It also outperforms many methods on out-of-distribution detection.

References in corpus (16)

• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Understanding Black-box Predictions via Influence Functions
• Weight Uncertainty in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Evidential Deep Learning to Quantify Classification Uncertainty
• Stochastic Gradient Hamiltonian Monte Carlo
• Natural Adversarial Examples
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• A Swiss Army Infinitesimal Jackknife
• The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning
• Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables
• A Higher-Order Swiss Army Infinitesimal Jackknife