Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes

Abstract

Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable covariance. We generalize linear response methods from statistical physics to deliver accurate uncertainty estimates for model variables---both for individual variables and coherently across variables. We call our method linear response variational Bayes (LRVB). When the MFVB posterior approximation is in the exponential family, LRVB has a simple, analytic form, even for non-conjugate models. Indeed, we make no assumptions about the form of the true posterior. We demonstrate the accuracy and scalability of our method on a range of models for both simulated and real data.

21 pages. arXiv admin note: substantial text overlap with arXiv:1502.07685

Cited by in corpus (10)

• Variational Inference: A Review for Statisticians
• Conditionally conjugate mean-field variational Bayes for logistic models
• Hierarchical Variational Models
• Fast and Accurate Estimation of Non-Nested Binomial Hierarchical Models Using Variational Inference
• Bayesian Weighted Mendelian Randomization for Causal Inference based on Summary Statistics
• Approximate Inference for Constructing Astronomical Catalogs from Images
• Fast robustness quantification with variational Bayes
• Variational Bayesian Approach and Gauss-Markov-Potts prior model
• Improving variational methods via pairwise linear response identities
• Efficient Semi-Implicit Variational Inference