Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach

Abstract

Bayesian inference typically requires the computation of an approximation to the posterior distribution. An important requirement for an approximate Bayesian inference algorithm is to output high-accuracy posterior mean and uncertainty estimates. Classical Monte Carlo methods, particularly Markov Chain Monte Carlo, remain the gold standard for approximate Bayesian inference because they have a robust finite-sample theory and reliable convergence diagnostics. However, alternative methods, which are more scalable or apply to problems where Markov Chain Monte Carlo cannot be used, lack the same finite-data approximation theory and tools for evaluating their accuracy. In this work, we develop a flexible new approach to bounding the error of mean and uncertainty estimates of scalable inference algorithms. Our strategy is to control the estimation errors in terms of Wasserstein distance, then bound the Wasserstein distance via a generalized notion of Fisher distance. Unlike computing the Wasserstein distance, which requires access to the normalized posterior distribution, the Fisher distance is tractable to compute because it requires access only to the gradient of the log posterior density. We demonstrate the usefulness of our Fisher distance approach by deriving bounds on the Wasserstein error of the Laplace approximation and Hilbert coresets. We anticipate that our approach will be applicable to many other approximate inference methods such as the integrated Laplace approximation, variational inference, and approximate Bayesian computation

arXiv comment22 pages, 2 figures

References in corpus (9)

• Frequentist Consistency of Variational Bayes
• Coresets for Scalable Bayesian Logistic Regression
• Robust and Scalable Bayes via a Median of Subset Posterior Measures
• Measuring Sample Quality with Kernels
• Automatic Variational Inference in Stan
• Quantitative bounds for Markov chain convergence: Wasserstein and total variation distances
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• (Non-) asymptotic properties of Stochastic Gradient Langevin Dynamics
• PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference

Cited by in corpus (7)

• User-friendly introduction to PAC-Bayes bounds
• Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Variational approximations using Fisher divergence
• Bayesian subset selection and variable importance for interpretable prediction and classification
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization
• Confidently Comparing Estimators with the c-value