The Case for Bayesian Deep Learning

Abstract

The key distinguishing property of a Bayesian approach is marginalization instead of optimization, not the prior, or Bayes rule. Bayesian inference is especially compelling for deep neural networks. (1) Neural networks are typically underspecified by the data, and can represent many different but high performing models corresponding to different settings of parameters, which is exactly when marginalization will make the biggest difference for both calibration and accuracy. (2) Deep ensembles have been mistaken as competing approaches to Bayesian methods, but can be seen as approximate Bayesian marginalization. (3) The structure of neural networks gives rise to a structured prior in function space, which reflects the inductive biases of neural networks that help them generalize. (4) The observed correlation between parameters in flat regions of the loss and a diversity of solutions that provide good generalization is further conducive to Bayesian marginalization, as flat regions occupy a large volume in a high dimensional space, and each different solution will make a good contribution to a Bayesian model average. (5) Recent practical advances for Bayesian deep learning provide improvements in accuracy and calibration compared to standard training, while retaining scalability.

References in corpus (7)

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
• Noise Contrastive Priors for Functional Uncertainty
• Bayesian GAN
• Understanding Generalization through Visualizations
• Output-Constrained Bayesian Neural Networks

Cited by in corpus (29)

• Global canopy height regression and uncertainty estimation from GEDI LIDAR waveforms with deep ensembles
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Logic Explained Networks
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Explaining Bayesian Neural Networks
• Towards More Practical Adversarial Attacks on Graph Neural Networks
• Cold Posteriors and Aleatoric Uncertainty
• Bayesian Graph Neural Networks for Molecular Property Prediction
• Bayesian Inference on Binary Spiking Networks Leveraging Nanoscale Device Stochasticity
• Theoretical characterization of uncertainty in high-dimensional linear classification
• Incorporating Interpretable Output Constraints in Bayesian Neural Networks
• Precise characterization of the prior predictive distribution of deep ReLU networks
• Bayesian sparsification for deep neural networks with Bayesian model reduction
• Meaningful uncertainties from deep neural network surrogates of large-scale numerical simulations
• Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning
• Efficient Online Bayesian Inference for Neural Bandits
• A Modulation Layer to Increase Neural Network Robustness Against Data Quality Issues
• Low Complexity Approximate Bayesian Logistic Regression for Sparse Online Learning
• Hessian-Free High-Resolution Nesterov Acceleration for Sampling
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• Methods for Recovering Conditional Independence Graphs: A Survey
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Scalable Bayesian neural networks by layer-wise input augmentation
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Rapid Risk Minimization with Bayesian Models Through Deep Learning Approximation
• Differentially private training of neural networks with Langevin dynamics for calibrated predictive uncertainty