Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit

Abstract

Modern deep learning models have achieved great success in predictive accuracy for many data modalities. However, their application to many real-world tasks is restricted by poor uncertainty estimates, such as overconfidence on out-of-distribution (OOD) data and ungraceful failing under distributional shift. Previous benchmarks have found that ensembles of neural networks (NNs) are typically the best calibrated models on OOD data. Inspired by this, we leverage recent theoretical advances that characterize the function-space prior of an ensemble of infinitely-wide NNs as a Gaussian process, termed the neural network Gaussian process (NNGP). We use the NNGP with a softmax link function to build a probabilistic model for multi-class classification and marginalize over the latent Gaussian outputs to sample from the posterior. This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue. We also examine the calibration of previous approaches to classification with the NNGP, which treat classification problems as regression to the one-hot labels. In this case the Bayesian posterior is exact, and we compare several heuristics to generate a categorical distribution over classes. We find these methods are well calibrated under distributional shift. Finally, we consider an infinite-width final layer in conjunction with a pre-trained embedding. This replicates the important practical use case of transfer learning and allows scaling to significantly larger datasets. As well as achieving competitive predictive accuracy, this approach is better calibrated than its finite width analogue.

arXiv comment23 pages, 11 figures

References in corpus (24)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• mixup: Beyond Empirical Risk Minimization
• On Calibration of Modern Neural Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Elliptical slice sampling
• Exponential expressivity in deep neural networks through transient chaos
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• Stochastic Variational Deep Kernel Learning
• Enhanced Convolutional Neural Tangent Kernels
• The Case for Bayesian Deep Learning
• The large learning rate phase of deep learning: the catapult mechanism
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• A Mean Field Theory of Batch Normalization
• Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Neural Kernels Without Tangents
• On the Importance of Strong Baselines in Bayesian Deep Learning
• Calibrating Deep Convolutional Gaussian Processes
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes

Cited by in corpus (1)

• Uncertainty Quantification From Scaling Laws in Deep Neural Networks