Laplace Redux -- Effortless Bayesian Deep Learning
arXiv:2106.14806
Abstract
Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection. The Laplace approximation (LA) is a classic, and arguably the simplest family of approximations for the intractable posteriors of deep neural networks. Yet, despite its simplicity, the LA is not as popular as alternatives like variational Bayes or deep ensembles. This may be due to assumptions that the LA is expensive due to the involved Hessian computation, that it is difficult to implement, or that it yields inferior results. In this work we show that these are misconceptions: we (i) review the range of variants of the LA including versions with minimal cost overhead; (ii) introduce "laplace", an easy-to-use software library for PyTorch offering user-friendly access to all major flavors of the LA; and (iii) demonstrate through extensive experiments that the LA is competitive with more popular alternatives in terms of performance, while excelling in terms of computational cost. We hope that this work will serve as a catalyst to a wider adoption of the LA in practical deep learning, including in domains where Bayesian approaches are not typically considered at the moment.
arXiv commentNeurIPS 2021 camera-ready version; source code: https://github.com/AlexImmer/Laplace
References in corpus (35)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Overcoming catastrophic forgetting in neural networks
- On Calibration of Modern Neural Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Scalable Bayesian Optimization Using Deep Neural Networks
- Accurate Uncertainties for Deep Learning Using Calibrated Regression
- New insights and perspectives on the natural gradient method
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Bayesian Dark Knowledge
- Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
- Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
- Variational inference for large-scale models of discrete choice
- Practical Deep Learning with Bayesian Principles
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
- On Quadratic Penalties in Elastic Weight Consolidation
- What Are Bayesian Neural Network Posteriors Really Like?
- Functional Regularisation for Continual Learning with Gaussian Processes
- Continual Deep Learning by Functional Regularisation of Memorable Past
- Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
- Practical Gauss-Newton Optimisation for Deep Learning
- How Good is the Bayes Posterior in Deep Neural Networks Really?
- Kernel Topic Models
- Adversarial Distillation of Bayesian Neural Network Posteriors
- 'In-Between' Uncertainty in Bayesian Neural Networks
- Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
- Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
- Subspace Inference for Bayesian Deep Learning
- Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks
- Fast Predictive Uncertainty for Classification with Bayesian Deep Networks
- Bayesian Optimization Meets Laplace Approximation for Robotic Introspection
- Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning