Variational Bayes for high-dimensional linear regression with sparse priors

Abstract

We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse truth at the optimal rate and gives optimal prediction of the response vector. The empirical performance of our algorithm is studied, showing that it works comparably well as other state-of-the-art Bayesian variable selection methods. We also numerically demonstrate that the widely used coordinate-ascent variational inference (CAVI) algorithm can be highly sensitive to the parameter updating order, leading to potentially poor performance. To mitigate this, we propose a novel prioritized updating scheme that uses a data-driven updating order and performs better in simulations. The variational algorithm is implemented in the R package 'sparsevb'.

42 pages. To appear in Journal of the American Statistical Association

References in corpus (13)

• Variational Inference: A Review for Statisticians
• Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
• Bayesian linear regression with sparse priors
• Microarrays, Empirical Bayes and the Two-Groups Model
• Needles and Straw in a Haystack: Posterior concentration for possibly sparse sequences
• Frequentist Consistency of Variational Bayes
• Empirical Bayes posterior concentration in sparse high-dimensional linear models
• Adaptive Bernstein-von Mises theorems in Gaussian white noise
• Nonparametric statistical inference for drift vector fields of multi-dimensional diffusions
• Bayesian Sparse Linear Regression with Unknown Symmetric Error
• Empirical priors for prediction in sparse high-dimensional linear regression
• Variational approximations of empirical Bayes posteriors in high-dimensional linear models
• In Search of Lost (Mixing) Time: Adaptive Markov chain Monte Carlo schemes for Bayesian variable selection with very large p

Cited by in corpus (7)

• Online Bayesian changepoint detection for network Poisson processes with community structure
• Variational inference based on a subclass of closed skew normals
• Adaptive variational Bayes: Optimality, computation and applications
• Variational Bayes latent class approach for EHR-based phenotyping with large real-world data
• Group Spike and Slab Variational Bayes
• Covariate-Elaborated Robust Partial Information Transfer with Conditional Spike-and-Slab Prior
• Optimal low-rank posterior mean and distribution approximation in linear Gaussian inverse problems on Hilbert spaces