PAC${}^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime

Abstract

The Bayesian posterior minimizes the "inferential risk" which itself bounds the "predictive risk". This bound is tight when the likelihood and prior are well-specified. However since misspecification induces a gap, the Bayesian posterior predictive distribution may have poor generalization performance. This work develops a multi-sample loss (PAC${}^m$) which can close the gap by spanning a trade-off between the two risks. The loss is computationally favorable and offers PAC generalization guarantees. Empirical study demonstrates improvement to the predictive distribution.

Accepted at AISTATS2022

References in corpus (6)

• Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealing
• Joint Distributions for TensorFlow Probability
• Automatic Differentiation Variational Inference with Mixtures