Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation
arXiv:2304.07048
Abstract
PAC-Bayes learning is an established framework to both assess the generalisation ability of learning algorithms, and design new learning algorithm by exploiting generalisation bounds as training objectives. Most of the exisiting bounds involve a \emph{Kullback-Leibler} (KL) divergence, which fails to capture the geometric properties of the loss function which are often useful in optimisation. We address this by extending the emerging \emph{Wasserstein PAC-Bayes} theory. We develop new PAC-Bayes bounds with Wasserstein distances replacing the usual KL, and demonstrate that sound optimisation guarantees translate to good generalisation abilities. In particular we provide generalisation bounds for the \emph{Bures-Wasserstein SGD} by exploiting its optimisation properties.
References in corpus (17)
- Generalization in Adaptive Data Analysis and Holdout Reuse
- On the properties of variational approximations of Gibbs posteriors
- Simpler PAC-Bayesian Bounds for Hostile Data
- Tighter risk certificates for neural networks
- PAC-Bayes under potentially heavy tails
- Variational inference via Wasserstein gradient flows
- Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning
- PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingales
- Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networks
- PAC-Bayes Un-Expected Bernstein Inequality
- Learning PAC-Bayes Priors for Probabilistic Neural Networks
- Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation
- Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes
- Integral Probability Metrics PAC-Bayes Bounds
- PAC-Bayesian Generalization Bounds for Adversarial Generative Models
- Tighter PAC-Bayes Bounds Through Coin-Betting
- Shedding a PAC-Bayesian Light on Adaptive Sliced-Wasserstein Distances