Simpler PAC-Bayesian Bounds for Hostile Data

Abstract

PAC-Bayesian learning bounds are of the utmost interest to the learning community. Their role is to connect the generalization ability of an aggregation distribution $\rho$ to its empirical risk and to its Kullback-Leibler divergence with respect to some prior distribution $\pi$. Unfortunately, most of the available bounds typically rely on heavy assumptions such as boundedness and independence of the observations. This paper aims at relaxing these constraints and provides PAC-Bayesian learning bounds that hold for dependent, heavy-tailed observations (hereafter referred to as \emph{hostile data}). In these bounds the Kullack-Leibler divergence is replaced with a general version of Csiszár's $f$-divergence. We prove a general PAC-Bayesian bound, and show how to use it in various hostile settings.

18 pages

References in corpus (6)

• Concentration inequalities for dependent Random variables via the martingale method
• Fast learning rates in statistical inference through aggregation
• On the properties of variational approximations of Gibbs posteriors
• PAC-Bayesian bounds for the Gram matrix and least squares regression with a random design
• Fast learning rates with heavy-tailed losses
• Learning from MOM's principles: Le Cam's approach

Cited by in corpus (26)

• User-friendly introduction to PAC-Bayes bounds
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• A Primer on PAC-Bayesian Learning
• PAC-Bayes under potentially heavy tails
• On the role of data in PAC-Bayes bounds
• Learning under Model Misspecification: Applications to Variational and Ensemble methods
• High dimensional VAR with low rank transition
• Data-dependent PAC-Bayes priors via differential privacy
• Stopping Criterion for Active Learning Based on Error Stability
• Exponential inequalities for nonstationary Markov Chains
• Meta-Learning PAC-Bayes Priors in Model Averaging
• Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation
• PAC-Bayesian Contrastive Unsupervised Representation Learning
• Bayesian Uncertainty Quantification for Anaerobic Digestion models
• PAC-Bayes Analysis Beyond the Usual Bounds
• Still no free lunches: the price to pay for tighter PAC-Bayes bounds
• Efron-Stein PAC-Bayesian Inequalities
• Non-exponentially weighted aggregation: regret bounds for unbounded loss functions
• PAC-Bayes unleashed: generalisation bounds with unbounded losses
• Learning via Wasserstein-Based High Probability Generalisation Bounds
• How Tight Can PAC-Bayes be in the Small Data Regime?
• Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation
• Unifying Variational Inference and PAC-Bayes for Supervised Learning that Scales
• PAC-Bayes-Chernoff bounds for unbounded losses
• Comparing Comparators in Generalization Bounds
• PAC-Bayesian bounds for learning LTI-ss systems with input from empirical loss