Bayesian Deep Learning via Subnetwork Inference
arXiv:2010.14689
Abstract
The Bayesian paradigm has the potential to solve core issues of deep neural networks such as poor calibration and data inefficiency. Alas, scaling Bayesian inference to large weight spaces often requires restrictive approximations. In this work, we show that it suffices to perform inference over a small subset of model weights in order to obtain accurate predictive posteriors. The other weights are kept as point estimates. This subnetwork inference framework enables us to use expressive, otherwise intractable, posterior approximations over such subsets. In particular, we implement subnetwork linearized Laplace as a simple, scalable Bayesian deep learning method: We first obtain a MAP estimate of all weights and then infer a full-covariance Gaussian posterior over a subnetwork using the linearized Laplace approximation. We propose a subnetwork selection strategy that aims to maximally preserve the model's predictive uncertainty. Empirically, our approach compares favorably to ensembles and less expressive posterior approximations over full networks. Our proposed subnetwork (linearized) Laplace method is implemented within the laplace PyTorch library at https://github.com/AlexImmer/Laplace.
arXiv commentICML 2021; 22 pages, extended version with supplementary material
References in corpus (33)
- Overcoming catastrophic forgetting in neural networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- On Calibration of Modern Neural Networks
- Weight Uncertainty in Neural Networks
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- Scalable Bayesian Optimization Using Deep Neural Networks
- Deep Ensembles: A Loss Landscape Perspective
- New insights and perspectives on the natural gradient method
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Practical Deep Learning with Bayesian Principles
- Functional Variational Bayesian Neural Networks
- What Are Bayesian Neural Network Posteriors Really Like?
- Deep Convolutional Networks as shallow Gaussian Processes
- Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
- Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
- Bayesian Batch Active Learning as Sparse Subset Approximation
- Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
- 'In-Between' Uncertainty in Bayesian Neural Networks
- Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
- Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
- Laplace Redux -- Effortless Bayesian Deep Learning
- SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
- Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
- The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
- Subspace Inference for Bayesian Deep Learning
- Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
- Benchmarking the Neural Linear Model for Regression
- Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
- Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning