Frequentist Consistency of Variational Bayes

Abstract

A key challenge for modern Bayesian statistics is how to perform scalable inference of posterior distributions. To address this challenge, variational Bayes (VB) methods have emerged as a popular alternative to the classical Markov chain Monte Carlo (MCMC) methods. VB methods tend to be faster while achieving comparable predictive performance. However, there are few theoretical results around VB. In this paper, we establish frequentist consistency and asymptotic normality of VB methods. Specifically, we connect VB methods to point estimates based on variational approximations, called frequentist variational approximations, and we use the connection to prove a variational Bernstein-von Mises theorem. The theorem leverages the theoretical characterizations of frequentist variational approximations to understand asymptotic properties of VB. In summary, we prove that (1) the VB posterior converges to the Kullback-Leibler (KL) minimizer of a normal distribution, centered at the truth and (2) the corresponding variational expectation of the parameter is consistent and asymptotically normal. As applications of the theorem, we derive asymptotic properties of VB posteriors in Bayesian mixture models, Bayesian generalized linear mixed models, and Bayesian stochastic block models. We conduct a simulation study to illustrate these theoretical results.

References in corpus (9)

• A Bayesian Approach to Network Modularity
• On the properties of variational approximations of Gibbs posteriors
• Asymptotic normality and valid inference for Gaussian variational approximation
• A Bernstein-Von Mises Theorem for discrete probability distributions
• The Bernstein--von Mises theorem for the proportional hazard model
• Large sample asymptotics for the two-parameter Poisson--Dirichlet process
• An Instability in Variational Inference for Topic Models
• On the Bernstein-Von Mises Theorem for High Dimensional Nonlinear Bayesian Inverse Problems
• On the use of bootstrap with variational inference: Theory, interpretation, and a two-sample test example

Cited by in corpus (60)

• Variational Bayes for high-dimensional linear regression with sparse priors
• Covariances, Robustness, and Variational Bayes
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• Bayesian Estimation of Mixed Multinomial Logit Models: Advances and Simulation-Based Evaluations
• Gibbs sampler and coordinate ascent variational inference: a set-theoretical review
• Advances in Variational Inference
• Variational Bayesian Approximation of Inverse Problems using Sparse Precision Matrices
• Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
• Approximating Bayes in the 21st Century
• Evaluating uncertainties in electrochemical impedance spectra of solid oxide fuel cells
• Desiderata for Representation Learning: A Causal Perspective
• Gaussian variational approximation for high-dimensional state space models
• Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
• Robust, Accurate Stochastic Optimization for Variational Inference
• Consistency of ELBO maximization for model selection
• Dynamics of coordinate ascent variational inference: A case study in 2D Ising models
• A Generalization Bound for Online Variational Inference
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Spike and slab variational Bayes for high dimensional logistic regression
• Bayesian Model Selection via Mean-Field Variational Approximation
• Convergence Rates of Variational Posterior Distributions
• Online Bayesian changepoint detection for network Poisson processes with community structure
• Validated Variational Inference via Practical Posterior Error Bounds
• Variational Inference in high-dimensional linear regression
• Statistical Inference in Mean-Field Variational Bayes
• Variational approximations of empirical Bayes posteriors in high-dimensional linear models
• Markov Switching
• Variational inference based on a subclass of closed skew normals
• An end-to-end Differentially Private Latent Dirichlet Allocation Using a Spectral Algorithm
• An approach to large-scale Quasi-Bayesian inference with spike-and-slab priors
• On the use of bootstrap with variational inference: Theory, interpretation, and a two-sample test example
• Variational Bayes under Model Misspecification
• Non-centered parametric variational Bayes' approach for hierarchical inverse problems of partial differential equations
• Local convexity of the TAP free energy and AMP convergence for Z2-synchronization
• Testing whether a Learning Procedure is Calibrated
• Frequentist Consistency of Generalized Variational Inference
• Sparse Variational Inference: Bayesian Coresets from Scratch
• Non-exponentially weighted aggregation: regret bounds for unbounded loss functions
• On the Robustness to Misspecification of $\alpha$-Posteriors and Their Variational Approximations
• Partially Observable Online Change Detection via Smooth-Sparse Decomposition
• Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables
• Bayesian Joint Chance Constrained Optimization: Approximations and Statistical Consistency
• Statistical optimality and stability of tangent transform algorithms in logit models
• Variational Discriminant Analysis with Variable Selection
• Bayesian Allocation Model: Inference by Sequential Monte Carlo for Nonnegative Tensor Factorizations and Topic Models using Polya Urns
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network
• A fast asynchronous MCMC sampler for sparse Bayesian inference
• When random initializations help: a study of variational inference for community detection
• PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models
• Simultaneous inference of periods and period-luminosity relations for Mira variable stars
• Variational Bayesian Methods for Stochastically Constrained System Design Problems
• On the Statistical Consistency of Risk-Sensitive Bayesian Decision-Making
• Distribution-Dependent Analysis of Gibbs-ERM Principle
• Incorporating prior information and borrowing information in high-dimensional sparse regression using the horseshoe and variational Bayes
• Convergence Rates of Empirical Bayes Posterior Distributions: A Variational Perspective
• Direct loss minimization algorithms for sparse Gaussian processes
• Fast, universal estimation of latent variable models using extended variational approximations
• Approximation Properties of Variational Bayes for Vector Autoregressions
• Variational Bayes Neural Network: Posterior Consistency, Classification Accuracy and Computational Challenges
• Exclusive Topic Modeling