What Are Bayesian Neural Network Posteriors Really Like?

Abstract

The posterior over Bayesian neural network (BNN) parameters is extremely high-dimensional and non-convex. For computational reasons, researchers approximate this posterior using inexpensive mini-batch methods such as mean-field variational inference or stochastic-gradient Markov chain Monte Carlo (SGMCMC). To investigate foundational questions in Bayesian deep learning, we instead use full-batch Hamiltonian Monte Carlo (HMC) on modern architectures. We show that (1) BNNs can achieve significant performance gains over standard training and deep ensembles; (2) a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains; (3) in contrast to recent studies, we find posterior tempering is not needed for near-optimal performance, with little evidence for a "cold posterior" effect, which we show is largely an artifact of data augmentation; (4) BMA performance is robust to the choice of prior scale, and relatively similar for diagonal Gaussian, mixture of Gaussian, and logistic priors; (5) Bayesian neural networks show surprisingly poor generalization under domain shift; (6) while cheaper alternatives such as deep ensembles and SGMCMC methods can provide good generalization, they provide distinct predictive distributions from HMC. Notably, deep ensemble predictive distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference.

References in corpus (31)

• Overcoming catastrophic forgetting in neural networks
• Gaussian Error Linear Units (GELUs)
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• On Calibration of Modern Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Searching for Activation Functions
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Deep Ensembles: A Loss Landscape Perspective
• Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• A Complete Recipe for Stochastic Gradient MCMC
• Essentially No Barriers in Neural Network Energy Landscape
• Practical Deep Learning with Bayesian Principles
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Bayesian Neural Network Priors Revisited
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• The Fundamental Incompatibility of Hamiltonian Monte Carlo and Data Subsampling
• Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting
• tfp.mcmc: Modern Markov Chain Monte Carlo Tools Built for Modern Hardware
• PBODL : Parallel Bayesian Online Deep Learning for Click-Through Rate Prediction in Tencent Advertising System
• A Bayesian neural network predicts the dissolution of compact planetary systems

Cited by in corpus (32)

• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• The Dark Energy Survey 5-year photometrically identified Type Ia Supernovae
• Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
• Laplace Redux -- Effortless Bayesian Deep Learning
• Asymptotics of representation learning in finite Bayesian neural networks
• Bayesian Neural Network Priors Revisited
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Repulsive Deep Ensembles are Bayesian
• Bayesian Deep Learning via Subnetwork Inference
• Exact marginal prior distributions of finite Bayesian neural networks
• On Sequential Bayesian Inference for Continual Learning
• Dangers of Bayesian Model Averaging under Covariate Shift
• Calibrating Bayesian Generative Machine Learning for Bayesiamplification
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Gibbs Sampling the Posterior of Neural Networks
• A Priori Uncertainty Quantification of Reacting Turbulence Closure Models using Bayesian Neural Networks
• Bayesian posterior approximation with stochastic ensembles
• Informative Bayesian Neural Network Priors for Weak Signals
• Bayesian OOD detection with aleatoric uncertainty and outlier exposure
• Epistemic Neural Networks
• Robust quantum dots charge autotuning using neural network uncertainty
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• Leveraging Active Subspaces to Capture Epistemic Model Uncertainty in Deep Generative Models for Molecular Design
• Looking at the posterior: accuracy and uncertainty of neural-network predictions
• Neural Variational Gradient Descent
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model
• Weight Pruning and Uncertainty in Radio Galaxy Classification
• Structured Stochastic Gradient MCMC
• Langevin Monte Carlo for strongly log-concave distributions: Randomized midpoint revisited
• Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference
• Data Subsampling for Bayesian Neural Networks
• Posterior Temperature Optimization in Variational Inference for Inverse Problems