Variational Inference in high-dimensional linear regression
arXiv:2104.12232
Abstract
We study high-dimensional Bayesian linear regression with product priors. Using the nascent theory of non-linear large deviations (Chatterjee and Dembo,2016), we derive sufficient conditions for the leading-order correctness of the naive mean-field approximation to the log-normalizing constant of the posterior distribution. Subsequently, assuming a true linear model for the observed data, we derive a limiting infinite dimensional variational formula for the log normalizing constant of the posterior. Furthermore, we establish that under an additional "separation" condition, the variational problem has a unique optimizer, and this optimizer governs the probabilistic properties of the posterior distribution. We provide intuitive sufficient conditions for the validity of this "separation" condition. Finally, we illustrate our results on concrete examples with specific design matrices.
39 pages
References in corpus (11)
- Spike and slab variable selection: Frequentist and Bayesian strategies
- Asymptotic normality of maximum likelihood and its variational approximation for stochastic blockmodels
- Frequentist Consistency of Variational Bayes
- Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression
- Nearly optimal Bayesian Shrinkage for High Dimensional Regression
- Asymptotic normality and valid inference for Gaussian variational approximation
- Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
- An Instability in Variational Inference for Topic Models
- Spike and slab variational Bayes for high dimensional logistic regression
- Statistical Inference in Mean-Field Variational Bayes
- The Mean-Field Approximation: Information Inequalities, Algorithms, and Complexity