PAC-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime
arXiv:2010.09629
Abstract
The Bayesian posterior minimizes the "inferential risk" which itself bounds the "predictive risk". This bound is tight when the likelihood and prior are well-specified. However since misspecification induces a gap, the Bayesian posterior predictive distribution may have poor generalization performance. This work develops a multi-sample loss (PAC) which can close the gap by spanning a trade-off between the two risks. The loss is computationally favorable and offers PAC generalization guarantees. Empirical study demonstrates improvement to the predictive distribution.
arXiv commentAccepted at AISTATS2022
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Deep Variational Information Bottleneck
- Variational inference for Monte Carlo objectives
- Generalized Variational Inference: Three arguments for deriving new Posteriors
- How Good is the Bayes Posterior in Deep Neural Networks Really?
- On Posterior Concentration in Misspecified Models
- Tighter Variational Bounds are Not Necessarily Better
- Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealing
- Measuring the reliability of MCMC inference with bidirectional Monte Carlo
- Joint Distributions for TensorFlow Probability
- Automatic Differentiation Variational Inference with Mixtures