A statistical theory of cold posteriors in deep neural networks

Abstract

To get Bayesian neural networks to perform comparably to standard neural networks it is usually necessary to artificially reduce uncertainty using a "tempered" or "cold" posterior. This is extremely concerning: if the prior is accurate, Bayes inference/decision theory is optimal, and any artificial changes to the posterior should harm performance. While this suggests that the prior may be at fault, here we argue that in fact, BNNs for image classification use the wrong likelihood. In particular, standard image benchmark datasets such as CIFAR-10 are carefully curated. We develop a generative model describing curation which gives a principled Bayesian account of cold posteriors, because the likelihood under this new generative model closely matches the tempered likelihoods used in past work.

Published at ICLR 2021 (https://openreview.net/forum?id=Rd138pWXMvG)

References in corpus (4)

• Scalable Variational Gaussian Process Classification
• L2 Regularization versus Batch and Weight Normalization
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Cold Posteriors and Aleatoric Uncertainty

Cited by in corpus (4)

• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Variational Laplace for Bayesian neural networks