Theoretical characterization of uncertainty in high-dimensional linear classification

Abstract

Being able to reliably assess not only the \emph{accuracy} but also the \emph{uncertainty} of models' predictions is an important endeavour in modern machine learning. Even if the model generating the data and labels is known, computing the intrinsic uncertainty after learning the model from a limited number of samples amounts to sampling the corresponding posterior probability measure. Such sampling is computationally challenging in high-dimensional problems and theoretical results on heuristic uncertainty estimators in high-dimensions are thus scarce. In this manuscript, we characterise uncertainty for learning from limited number of samples of high-dimensional Gaussian input data and labels generated by the probit model. In this setting, the Bayesian uncertainty (i.e. the posterior marginals) can be asymptotically obtained by the approximate message passing algorithm, bypassing the canonical but costly Monte Carlo sampling of the posterior. We then provide a closed-form formula for the joint statistics between the logistic classifier, the uncertainty of the statistically optimal Bayesian classifier and the ground-truth probit uncertainty. The formula allows us to investigate calibration of the logistic classifier learning from limited amount of samples. We discuss how over-confidence can be mitigated by appropriately regularising.

References in corpus (23)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• On Calibration of Modern Neural Networks
• A tutorial on conformal prediction
• Statistical physics of inference: Thresholds and algorithms
• Gaussian Processes for Big Data
• A modern maximum-likelihood theory for high-dimensional logistic regression
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• The Nishimori line and Bayesian Statistics
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Generalisation error in learning with random features and the hidden manifold model
• The Case for Bayesian Deep Learning
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization
• The committee machine: Computational to statistical gaps in learning a two-layers neural network
• Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Exact asymptotics for phase retrieval and compressed sensing with random generative priors
• Cold Posteriors and Aleatoric Uncertainty
• On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification
• A Parsimonious Tour of Bayesian Model Uncertainty