Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes

Abstract

We consider the optimal approximate posterior over the top-layer weights in a Bayesian neural network for regression, and show that it exhibits strong dependencies on the lower-layer weights. We adapt this result to develop a correlated approximate posterior over the weights at all layers in a Bayesian neural network. We extend this approach to deep Gaussian processes, unifying inference in the two model classes. Our approximate posterior uses learned "global" inducing points, which are defined only at the input layer and propagated through the network to obtain inducing inputs at subsequent layers. By contrast, standard, "local", inducing point methods from the deep Gaussian process literature optimise a separate set of inducing inputs at every layer, and thus do not model correlations across layers. Our method gives state-of-the-art performance for a variational Bayesian method, without data augmentation or tempering, on CIFAR-10 of 86.7%, which is comparable to SGMCMC without tempering but with data augmentation (88% in Wenzel et al. 2020).

arXiv commentAccepted for publication at the 38th International Conference on Machine Learning (ICML 2021, PMLR 139), 33 pages

References in corpus (34)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Deep Neural Networks as Gaussian Processes
• Variational Dropout and the Local Reparameterization Trick
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Practical Deep Learning with Bayesian Principles
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Enhanced Convolutional Neural Tangent Kernels
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• Non-Stationary Gaussian Process Regression with Hamiltonian Monte Carlo
• Implicit Weight Uncertainty in Neural Networks
• Convolutional Gaussian Processes
• Structured Stochastic Variational Inference
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Overpruning in Variational Bayesian Neural Networks
• Noisy Natural Gradient as Variational Inference
• Bayesian Neural Network Priors Revisited
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• Eigenvalue Corrected Noisy Natural Gradient
• A statistical theory of cold posteriors in deep neural networks
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Why bigger is not always better: on finite and infinite neural networks
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Importance Weighted Hierarchical Variational Inference
• Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties
• Deep kernel processes

Cited by in corpus (8)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Finite Versus Infinite Neural Networks: an Empirical Study
• Bayesian Neural Network Priors Revisited
• A statistical theory of cold posteriors in deep neural networks
• Deep kernel processes
• Variational Laplace for Bayesian neural networks
• A variational approximate posterior for the deep Wishart process
• Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning