Nearly Optimal Variational Inference for High Dimensional Regression with Shrinkage Priors

Abstract

We propose a variational Bayesian (VB) procedure for high-dimensional linear model inferences with heavy tail shrinkage priors, such as student-t prior. Theoretically, we establish the consistency of the proposed VB method and prove that under the proper choice of prior specifications, the contraction rate of the VB posterior is nearly optimal. It justifies the validity of VB inference as an alternative of Markov Chain Monte Carlo (MCMC) sampling. Meanwhile, comparing to conventional MCMC methods, the VB procedure achieves much higher computational efficiency, which greatly alleviates the computing burden for modern machine learning applications such as massive data analysis. Through numerical studies, we demonstrate that the proposed VB method leads to shorter computing time, higher estimation accuracy, and lower variable selection error than competitive sparse Bayesian methods.

References in corpus (7)

• Nearly unbiased variable selection under minimax concave penalty
• Spike and slab variable selection: Frequentist and Bayesian strategies
• Contributed Discussion to Uncertainty Quantification for the Horseshoe by Stéphanie van der Pas, Botond Szabó and Aad van der Vaart
• Posterior consistency in linear models under shrinkage priors
• Nearly optimal Bayesian Shrinkage for High Dimensional Regression
• A Variational Algorithm for Bayesian Variable Selection
• Posterior Concentration Properties of a General Class of Shrinkage Priors around Nearly Black Vectors

Cited by in corpus (1)

• Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey