PAC-Bayes under potentially heavy tails

Abstract

We derive PAC-Bayesian learning guarantees for heavy-tailed losses, and obtain a novel optimal Gibbs posterior which enjoys finite-sample excess risk bounds at logarithmic confidence. Our core technique itself makes use of PAC-Bayesian inequalities in order to derive a robust risk estimator, which by design is easy to compute. In particular, only assuming that the first three moments of the loss distribution are bounded, the learning algorithm derived from this estimator achieves nearly sub-Gaussian statistical error, up to the quality of the prior.

References in corpus (2)

• A Note on the PAC Bayesian Theorem
• Dimension-free PAC-Bayesian bounds for matrices, vectors, and linear least squares regression

Cited by in corpus (12)

• User-friendly introduction to PAC-Bayes bounds
• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence
• PAC-Bayes Analysis Beyond the Usual Bounds
• Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation
• Still no free lunches: the price to pay for tighter PAC-Bayes bounds
• Efron-Stein PAC-Bayesian Inequalities
• Upper and Lower Bounds on the Performance of Kernel PCA
• Non-exponentially weighted aggregation: regret bounds for unbounded loss functions
• PAC-Bayes unleashed: generalisation bounds with unbounded losses
• Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation
• PAC-Bayes-Chernoff bounds for unbounded losses