A statistical theory of cold posteriors in deep neural networks

Abstract

To get Bayesian neural networks to perform comparably to standard neural networks it is usually necessary to artificially reduce uncertainty using a "tempered" or "cold" posterior. This is extremely concerning: if the prior is accurate, Bayes inference/decision theory is optimal, and any artificial changes to the posterior should harm performance. While this suggests that the prior may be at fault, here we argue that in fact, BNNs for image classification use the wrong likelihood. In particular, standard image benchmark datasets such as CIFAR-10 are carefully curated. We develop a generative model describing curation which gives a principled Bayesian account of cold posteriors, because the likelihood under this new generative model closely matches the tempered likelihoods used in past work.

arXiv commentPublished at ICLR 2021 (https://openreview.net/forum?id=Rd138pWXMvG)

References in corpus (10)

• Galaxy Zoo 2: detailed morphological classifications for 304,122 galaxies from the Sloan Digital Sky Survey
• Scalable Variational Gaussian Process Classification
• L2 Regularization versus Batch and Weight Normalization
• Practical Deep Learning with Bayesian Principles
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning
• Noisy Natural Gradient as Variational Inference
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Cold Posteriors and Aleatoric Uncertainty
• Eigenvalue Corrected Noisy Natural Gradient

Cited by in corpus (4)

• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Variational Laplace for Bayesian neural networks