Bayesian variable selection with shrinking and diffusing priors

Abstract

We consider a Bayesian approach to variable selection in the presence of high dimensional covariates based on a hierarchical model that places prior distributions on the regression coefficients as well as on the model space. We adopt the well-known spike and slab Gaussian priors with a distinct feature, that is, the prior variances depend on the sample size through which appropriate shrinkage can be achieved. We show the strong selection consistency of the proposed method in the sense that the posterior probability of the true model converges to one even when the number of covariates grows nearly exponentially with the sample size. This is arguably the strongest selection consistency result that has been available in the Bayesian variable selection literature; yet the proposed method can be carried out through posterior sampling with a simple Gibbs sampler. Furthermore, we argue that the proposed method is asymptotically similar to model selection with the $L_0$ penalty. We also demonstrate through empirical work the fine performance of the proposed approach relative to some state of the art alternatives.

Published in at http://dx.doi.org/10.1214/14-AOS1207 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (8)

• Spike and slab variable selection: Frequentist and Bayesian strategies
• Nonconcave penalized likelihood with a diverging number of parameters
• Optimal predictive model selection
• Bayesian empirical likelihood for quantile regression
• Consistency of objective Bayes factors as the model dimension grows
• Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities
• Asymptotic oracle properties of SCAD-penalized least squares estimators
• Penalized Orthogonal-Components Regression for Large p Small n Data

Cited by in corpus (58)

• Comparison of Bayesian predictive methods for model selection
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Empirical Bayes posterior concentration in sparse high-dimensional linear models
• On spike-and-slab priors for Bayesian equation discovery of nonlinear dynamical systems via sparse linear regression
• High-Dimensional Multivariate Posterior Consistency Under Global-Local Shrinkage Priors
• Spike-and-Slab Meets LASSO: A Review of the Spike-and-Slab LASSO
• Weighted Bayesian Bootstrap for Scalable Bayes
• Scalable Bayesian Variable Selection Using Nonlocal Prior Densities in Ultrahigh-Dimensional Settings
• Projection Predictive Inference for Generalized Linear and Additive Multilevel Models
• On the Beta Prime Prior for Scale Parameters in High-Dimensional Bayesian Regression Models
• Bayesian Sparse Linear Regression with Unknown Symmetric Error
• Big Learning with Bayesian Methods
• Random weighting in LASSO regression
• Efficient estimation and correction of selection-induced bias with order statistics
• Bayesian inference in high-dimensional linear models using an empirical correlation-adaptive prior
• Concentration of posterior probabilities and normalized L0 criteria
• Variable Selection with ABC Bayesian Forests
• A Moreau-Yosida approximation scheme for a class of high-dimensional posterior distributions
• A Parsimonious Tour of Bayesian Model Uncertainty
• Posterior Graph Selection and Estimation Consistency for High-dimensional Bayesian DAG Models
• Neuronized Priors for Bayesian Sparse Linear Regression
• An approach to large-scale Quasi-Bayesian inference with spike-and-slab priors
• Probabilistic community detection with unknown number of communities
• Dimension-free Mixing for High-dimensional Bayesian Variable Selection
• Complexity analysis of Bayesian learning of high-dimensional DAG models and their equivalence classes
• Interaction pursuit in high-dimensional multi-response regression via distance correlation
• A Bayesian Approach to Joint Estimation of Multiple Graphical Models
• The EAS approach to variable selection for multivariate response data in high-dimensional settings
• Covariate-Assisted Bayesian Graph Learning for Heterogeneous Data
• Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes
• A Metropolized adaptive subspace algorithm for high-dimensional Bayesian variable selection
• Bayesian high-dimensional linear regression with generic spike-and-slab priors
• Bayesian equation selection on sparse data for discovery of stochastic dynamical systems
• Scalable Bayesian change point detection with spike and slab priors
• Risk and Regret of Hierarchical Bayesian Learners
• Minimax Posterior Convergence Rates and Model Selection Consistency in High-dimensional DAG Models based on Sparse Cholesky Factors
• Approximate spectral gaps for Markov chains mixing times in high dimensions
• Bayesian Analysis of High-dimensional Discrete Graphical Models
• Variable Selection Consistency of Gaussian Process Regression
• Gene-gene interaction analysis incorporating network information via a structured Bayesian approach
• Approximate Laplace approximations for scalable model selection
• Bayesian iterative screening in ultra-high dimensional linear regressions
• A robust Bayesian analysis of variable selection under prior ignorance
• Posterior consistency in multi-response regression models with non-informative priors for the error covariance matrix in growing dimensions
• Forecasting under model uncertainty:Non-homogeneous hidden Markov models with Polya-Gamma data augmentation
• Fully Bayesian Penalized Regression with a Generalized Bridge Prior
• Consistent Bayesian Sparsity Selection for High-dimensional Gaussian DAG Models with Multiplicative and Beta-mixture Priors
• Revisiting High Dimensional Bayesian Model Selection for Gaussian Regression
• Non-penalized variable selection in high-dimensional linear model settings via generalized fiducial inference
• Bayesian Edge Regression in Undirected Graphical Models to Characterize Interpatient Heterogeneity in Cancer
• Unified Bayesian theory of sparse linear regression with nuisance parameters
• On Posterior consistency of Bayesian Changepoint models
• Adaptive posterior convergence in sparse high dimensional clipped generalized linear models
• Bayesian High-dimensional Semi-parametric Inference beyond sub-Gaussian Errors
• Bayesian Group Selection in Logistic Regression with Application to MRI Data Analysis
• On Joint Estimation of Gaussian Graphical Models for Spatial and Temporal Data
• Bayesian Factor-adjusted Sparse Regression
• Covariate-Elaborated Robust Partial Information Transfer with Conditional Spike-and-Slab Prior