Improving predictions of Bayesian neural nets via local linearization
arXiv:2008.08400
Abstract
The generalized Gauss-Newton (GGN) approximation is often used to make practical Bayesian deep learning approaches scalable by replacing a second order derivative with a product of first order derivatives. In this paper we argue that the GGN approximation should be understood as a local linearization of the underlying Bayesian neural network (BNN), which turns the BNN into a generalized linear model (GLM). Because we use this linearized model for posterior inference, we should also predict using this modified model instead of the original one. We refer to this modified predictive as "GLM predictive" and show that it effectively resolves common underfitting problems of the Laplace approximation. It extends previous results in this vein to general likelihoods and has an equivalent Gaussian process formulation, which enables alternative inference schemes for BNNs in function space. We demonstrate the effectiveness of our approach on several standard classification datasets as well as on out-of-distribution detection. We provide an implementation at https://github.com/AlexImmer/BNN-predictions.
arXiv commentAISTATS 2021
References in corpus (15)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Weight Uncertainty in Neural Networks
- GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Variational Dropout and the Local Reparameterization Trick
- Scalable Variational Gaussian Process Classification
- New insights and perspectives on the natural gradient method
- Practical Deep Learning with Bayesian Principles
- Fast Variational Inference in the Conjugate Exponential Family
- Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
- Continual Deep Learning by Functional Regularisation of Memorable Past
- Conditional Neural Processes
- Practical Gauss-Newton Optimisation for Deep Learning
- How Good is the Bayes Posterior in Deep Neural Networks Really?
- 'In-Between' Uncertainty in Bayesian Neural Networks