Subspace Inference for Bayesian Deep Learning

Abstract

Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this paper, we construct low-dimensional subspaces of parameter space, such as the first principal components of the stochastic gradient descent (SGD) trajectory, which contain diverse sets of high performing models. In these subspaces, we are able to apply elliptical slice sampling and variational inference, which struggle in the full parameter space. We show that Bayesian model averaging over the induced posterior in these subspaces produces accurate predictions and well calibrated predictive uncertainty for both regression and image classification.

Published at UAI 2019

References in corpus (16)

• Weight Uncertainty in Neural Networks
• A Widely Applicable Bayesian Information Criterion
• Variational Dropout and the Local Reparameterization Trick
• Elliptical slice sampling
• Gaussian Process Kernels for Pattern Discovery and Extrapolation
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Gradient Descent Happens in a Tiny Subspace
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• Measuring the Intrinsic Dimension of Objective Landscapes
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Orthogonally Decoupled Variational Gaussian Processes
• Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions
• Bayesian Hypernetworks
• Probabilistic Meta-Representations Of Neural Networks

Cited by in corpus (10)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Laplace Redux -- Effortless Bayesian Deep Learning
• Bayesian Neural Ordinary Differential Equations
• Bayesian Deep Learning via Subnetwork Inference
• Bayesian Graph Neural Networks for Molecular Property Prediction
• Bayesian Methods for Semi-supervised Text Annotation
• Towards Expressive Priors for Bayesian Neural Networks: Poisson Process Radial Basis Function Networks
• URSABench: Comprehensive Benchmarking of Approximate Bayesian Inference Methods for Deep Neural Networks
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights