Quantifying Epistemic Uncertainty in Deep Learning

Abstract

Uncertainty quantification is at the core of the reliability and robustness of machine learning. In this paper, we provide a theoretical framework to dissect the uncertainty, especially the \textit{epistemic} component, in deep learning into \textit{procedural variability} (from the training procedure) and \textit{data variability} (from the training data), which is the first such attempt in the literature to our best knowledge. We then propose two approaches to estimate these uncertainties, one based on influence function and one on batching. We demonstrate how our approaches overcome the computational difficulties in applying classical statistical methods. Experimental evaluations on multiple problem settings corroborate our theory and illustrate how our framework and estimation can provide direct guidance on modeling and data collection efforts.

References in corpus (28)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding Black-box Predictions via Influence Functions
• A Convergence Theory for Deep Learning via Over-Parameterization
• On Exact Computation with an Infinitely Wide Neural Net
• Deep Ensembles: A Loss Landscape Perspective
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Batch means and spectral variance estimators in Markov chain Monte Carlo
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Approximating Continuous Functions by ReLU Nets of Minimal Width
• Consistency and robustness of kernel-based regression in convex risk minimization
• On the Convergence Rate of Training Recurrent Neural Networks
• On the Accuracy of Influence Functions for Measuring Group Effects
• Enhanced Convolutional Neural Tangent Kernels
• Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control
• Neural Contextual Bandits with UCB-based Exploration
• Nonparametric Conditional Density Estimation in a High-Dimensional Regression Setting
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Conditional Density Estimation Tools in Python and R with Applications to Photometric Redshifts and Likelihood-Free Cosmological Inference
• Neural Thompson Sampling
• Gaussian Process Conditional Density Estimation
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• A Unified Framework for Constructing, Tuning and Assessing Photometric Redshift Density Estimates in a Selection Bias Setting
• Influence Functions in Deep Learning Are Fragile
• Frequentist Uncertainty in Recurrent Neural Networks via Blockwise Influence Functions
• Understanding the Under-Coverage Bias in Uncertainty Estimation