Variational Laplace for Bayesian neural networks

Abstract

We develop variational Laplace for Bayesian neural networks (BNNs) which exploits a local approximation of the curvature of the likelihood to estimate the ELBO without the need for stochastic sampling of the neural-network weights. The Variational Laplace objective is simple to evaluate, as it is (in essence) the log-likelihood, plus weight-decay, plus a squared-gradient regularizer. Variational Laplace gave better test performance and expected calibration errors than maximum a-posteriori inference and standard sampling-based variational inference, despite using the same variational approximate posterior. Finally, we emphasise care needed in benchmarking standard VI as there is a risk of stopping before the variance parameters have converged. We show that early-stopping can be avoided by increasing the learning rate for the variance parameters.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• End to End Learning for Self-Driving Cars
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• Improving Explorability in Variational Inference with Annealed Variational Objectives
• The variational Laplace approach to approximate Bayesian inference
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• A statistical theory of cold posteriors in deep neural networks
• Implicit Gradient Regularization