How Good is the Bayes Posterior in Deep Neural Networks Really?

Abstract

During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there are---as of early 2020---no publicized deployments of Bayesian neural networks in industrial practice. In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD. Furthermore, we demonstrate that predictive performance is improved significantly through the use of a "cold posterior" that overcounts evidence. Such cold posteriors sharply deviate from the Bayesian paradigm but are commonly used as heuristic in Bayesian deep learning papers. We put forward several hypotheses that could explain cold posteriors and evaluate the hypotheses through experiments. Our work questions the goal of accurate posterior approximations in Bayesian deep learning: If the true Bayes posterior is poor, what is the use of more accurate approximations? Instead, we argue that it is timely to focus on understanding the origin of the improved performance of cold posteriors.

arXiv commentFull version (main paper and appendix) of the ICML 2020 publication

References in corpus (32)

• MCMC using Hamiltonian dynamics
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Deep Neural Networks as Gaussian Processes
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout and the Local Reparameterization Trick
• Revisiting Small Batch Training for Deep Neural Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Practical Deep Learning with Bayesian Principles
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Functional Variational Bayesian Neural Networks
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• The Case for Bayesian Deep Learning
• Noise Contrastive Priors for Functional Uncertainty
• Multi-Sample Dropout for Accelerated Training and Better Generalization
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Fluctuation-dissipation relations for stochastic gradient descent
• On Posterior Concentration in Misspecified Models
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Dropout as a Structured Shrinkage Prior
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• Variational Tempering
• tfp.mcmc: Modern Markov Chain Monte Carlo Tools Built for Modern Hardware
• Bayesian Inference for Large Scale Image Classification
• Covariance-Controlled Adaptive Langevin Thermostat for Large-Scale Bayesian Sampling
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Eigenvalue Corrected Noisy Natural Gradient
• Bayesian Neural Networks at Finite Temperature

Cited by in corpus (15)

• Probabilistic neural networks for fluid flow surrogate modeling and data recovery
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Laplace Redux -- Effortless Bayesian Deep Learning
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• How Much Can I Trust You? -- Quantifying Uncertainties in Explaining Neural Networks
• Cold Posteriors and Aleatoric Uncertainty
• Improving predictions of Bayesian neural nets via local linearization
• Stationary Activations for Uncertainty Calibration in Deep Learning
• Strength of Minibatch Noise in SGD
• Informative Bayesian Neural Network Priors for Weak Signals
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• PAC${}^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime
• Weight Pruning and Uncertainty in Radio Galaxy Classification
• A Bayesian Approach to Invariant Deep Neural Networks
• The Effect of Prior Lipschitz Continuity on the Adversarial Robustness of Bayesian Neural Networks