Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation

Abstract

We introduce several novel change of measure inequalities for two families of divergences: $f$-divergences and $\alpha$-divergences. We show how the variational representation for $f$-divergences leads to novel change of measure inequalities. We also present a multiplicative change of measure inequality for $\alpha$-divergences and a generalized version of Hammersley-Chapman-Robbins inequality. Finally, we present several applications of our change of measure inequalities, including PAC-Bayesian bounds for various classes of losses and non-asymptotic intervals for Monte Carlo estimates.

arXiv comment20 pages

References in corpus (6)

• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• A Primer on PAC-Bayesian Learning
• Simpler PAC-Bayesian Bounds for Hostile Data
• Scalable Information Inequalities for Uncertainty Quantification
• PAC-Bayes under potentially heavy tails
• Tighter Variational Representations of f-Divergences via Restriction to Probability Measures

Cited by in corpus (2)

• Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation
• Learning via Wasserstein-Based High Probability Generalisation Bounds