On the properties of variational approximations of Gibbs posteriors

Abstract

The PAC-Bayesian approach is a powerful set of techniques to derive non- asymptotic risk bounds for random estimators. The corresponding optimal distribution of estimators, usually called the Gibbs posterior, is unfortunately intractable. One may sample from it using Markov chain Monte Carlo, but this is often too slow for big datasets. We consider instead variational approximations of the Gibbs posterior, which are fast to compute. We undertake a general study of the properties of such approximations. Our main finding is that such a variational approximation has often the same rate of convergence as the original PAC-Bayesian procedure it approximates. We specialise our results to several learning tasks (classification, ranking, matrix completion),discuss how to implement a variational approximation in each case, and illustrate the good properties of said approximation on real datasets.

References in corpus (8)

• An MCMC Approach to Classical Estimation
• Sparse Regression Learning by Aggregation and Langevin Monte-Carlo
• Gibbs posterior for variable selection in high-dimensional classification and data mining
• Sparse single-index model
• A Stochastic Quasi-Newton Method for Large-Scale Optimization
• Bayesian methods for low-rank matrix estimation: short survey and theoretical study
• Bayesian computation: a perspective on the current state, and sampling backwards and forwards
• PAC-Bayesian AUC classification and scoring

Cited by in corpus (28)

• Frequentist Consistency of Variational Bayes
• Calibrating general posterior credible regions
• User-friendly introduction to PAC-Bayes bounds
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• Simpler PAC-Bayesian Bounds for Hostile Data
• 1-bit Matrix Completion: PAC-Bayesian Analysis of a Variational Approximation
• Tighter risk certificates for neural networks
• Gibbs posterior concentration rates under sub-exponential type losses
• Pseudo-Bayesian Quantum Tomography with Rank-adaptation
• Model-free posterior inference on the area under the receiver operating characteristic curve
• Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
• Empirical Risk Minimization with Relative Entropy Regularization
• Bayesian Model Selection via Mean-Field Variational Approximation
• From bilinear regression to inductive matrix completion: a quasi-Bayesian analysis
• Meta-Learning PAC-Bayes Priors in Model Averaging
• High-dimensional sparse classification using exponential weighting with empirical hinge loss
• Adaptive variational Bayes: Optimality, computation and applications
• PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNs
• PAC-Bayes unleashed: generalisation bounds with unbounded losses
• An efficient adaptive MCMC algorithm for Pseudo-Bayesian quantum tomography
• Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes
• Explicit construction of the minimum error variance estimator for stochastic LTI state-space systems
• Learning via Wasserstein-Based High Probability Generalisation Bounds
• Semiparametric inference using fractional posteriors
• Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation
• PAC-Bayes-Chernoff bounds for unbounded losses
• PAC-Bayesian bounds for learning LTI-ss systems with input from empirical loss
• A note on regularised NTK dynamics with an application to PAC-Bayesian training