PAC-Bayes-Chernoff bounds for unbounded losses
arXiv:2401.01148
Abstract
We introduce a new PAC-Bayes oracle bound for unbounded losses that extends Cramér-Chernoff bounds to the PAC-Bayesian setting. The proof technique relies on controlling the tails of certain random variables involving the Cramér transform of the loss. Our approach naturally leverages properties of Cramér-Chernoff bounds, such as exact optimization of the free parameter in many PAC-Bayes bounds. We highlight several applications of the main theorem. Firstly, we show that our bound recovers and generalizes previous results. Additionally, our approach allows working with richer assumptions that result in more informative and potentially tighter bounds. In this direction, we provide a general bound under a new \textit{model-dependent} assumption from which we obtain bounds based on parameter norms and log-Sobolev inequalities. Notably, many of these bounds can be minimized to obtain distributions beyond the Gibbs posterior and provide novel theoretical coverage to existing regularization techniques.
Camera-ready version for NeurIPS 2024
References in corpus (20)
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- A Note on the PAC Bayesian Theorem
- On the properties of variational approximations of Gibbs posteriors
- Generalization Bounds via Information Density and Conditional Information Density
- Simpler PAC-Bayesian Bounds for Hostile Data
- A Strongly Quasiconvex PAC-Bayesian Bound
- Bayesian Model Selection, the Marginal Likelihood, and Generalization
- PAC-Bayes under potentially heavy tails
- On strict sub-Gaussianity, optimal proxy variance and symmetry for bounded random variables
- Learning under Model Misspecification: Applications to Variational and Ensemble methods
- PAC-Bayes-Bernstein Inequality for Martingales and its Application to Multiarmed Bandits
- Second Order PAC-Bayesian Bounds for the Weighted Majority Vote
- Generalization Bounds: Perspectives from Information Theory and PAC-Bayes
- Upper and Lower Bounds on the Performance of Kernel PCA
- More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validity
- PAC-Chernoff Bounds: Understanding Generalization in the Interpolation Regime
- On the Importance of Gradient Norm in PAC-Bayesian Bounds
- Chebyshev-Cantelli PAC-Bayes-Bennett Inequality for the Weighted Majority Vote
- Comparing Comparators in Generalization Bounds
- Split-kl and PAC-Bayes-split-kl Inequalities for Ternary Random Variables