Spike and slab variable selection: Frequentist and Bayesian strategies

Abstract

Variable selection in the linear regression model takes many apparent faces from both frequentist and Bayesian standpoints. In this paper we introduce a variable selection method referred to as a rescaled spike and slab model. We study the importance of prior hierarchical specifications and draw connections to frequentist generalized ridge regression estimation. Specifically, we study the usefulness of continuous bimodal priors to model hypervariance parameters, and the effect scaling has on the posterior mean through its relationship to penalization. Several model selection strategies, some frequentist and some Bayesian in nature, are developed and studied theoretically. We demonstrate the importance of selective shrinkage for effective variable selection in terms of risk misclassification, and show this is achieved using the posterior from a rescaled spike and slab model. We also show how to verify a procedure's ability to reduce model uncertainty in finite samples using a specialized forward selection strategy. Using this tool, we illustrate the effectiveness of rescaled spike and slab models in reducing model uncertainty.

Published at http://dx.doi.org/10.1214/009053604000001147 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (11)

• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Optimal predictive model selection

Cited by in corpus (164)

• Bayesian linear regression with sparse priors
• Bayesian variable selection with shrinking and diffusing priors
• Variable selection for BART: An application to gene regulation
• Spike-and-Slab Priors for Function Selection in Structured Additive Regression Models
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• A Message-Passing Receiver for BICM-OFDM over Unknown Clustered-Sparse Channels
• Empirical Bayes posterior concentration in sparse high-dimensional linear models
• On spike-and-slab priors for Bayesian equation discovery of nonlinear dynamical systems via sparse linear regression
• An MDL framework for sparse coding and dictionary learning
• Nearly optimal Bayesian Shrinkage for High Dimensional Regression
• High-Dimensional Multivariate Posterior Consistency Under Global-Local Shrinkage Priors
• A note on the multiplicative gamma process
• Spike-and-Slab Meets LASSO: A Review of the Spike-and-Slab LASSO
• A majorization-minimization approach to variable selection using spike and slab priors
• Mapping unobserved item-respondent interactions: A latent space item response model with interaction map
• ICR: Iterative Convex Refinement for Sparse Signal Recovery Using Spike and Slab Priors
• Local stability and robustness of sparse dictionary learning in the presence of noise
• Bayesian and L1 Approaches to Sparse Unsupervised Learning
• Inferring the Direction of a Causal Link and Estimating Its Effect via a Bayesian Mendelian Randomization Approach
• Sparse Linear Identifiable Multivariate Modeling
• Scalable Bayesian Variable Selection Using Nonlocal Prior Densities in Ultrahigh-Dimensional Settings
• Robust Sonar ATR Through Bayesian Pose Corrected Sparse Classification
• Graph Learning from Data under Structural and Laplacian Constraints
• Deep Exponential Families
• Penalising model component complexity: A principled, practical approach to constructing priors
• Function-on-function kriging, with applications to 3D printing of aortic tissues
• A Bayesian Time-Varying Effect Model for Behavioral mHealth Data
• Exploring the determinants of Bitcoin's price: an application of Bayesian Structural Time Series
• Projection Predictive Inference for Generalized Linear and Additive Multilevel Models
• Efficient inference for genetic association studies with multiple outcomes
• Learning credit assignment
• Minimax Optimal Bayesian Aggregation
• Bayesian methods to overcome the winner's curse in genetic studies
• Fully Bayesian Logistic Regression with Hyper-Lasso Priors for High-dimensional Feature Selection
• Bayesian Sparse Linear Regression with Unknown Symmetric Error
• A Deep Generative Deconvolutional Image Model
• Thresholded Multiscale Gaussian Processes with Application to Bayesian Feature Selection for Massive Neuroimaging Data
• Bayesian Structured Sparsity from Gaussian Fields
• Big Learning with Bayesian Methods
• Connecting the Dots Between MLE and RL for Sequence Prediction
• Bayesian Effect Selection in Structured Additive Distributional Regression Models
• Adaptive Supremum Norm Posterior Contraction: Wavelet Spike-and-Slab and Anisotropic Besov Spaces
• Bayesian Model Selection via Mean-Field Variational Approximation
• Dynamic sparsity on dynamic regression models
• A new Bayesian regression model for counts in medicine
• Bayesian Learning of Parameterised Quantum Circuits
• On Oracle Property and Asymptotic Validity of Bayesian Generalized Method of Moments
• Bayesian compressed sensing with new sparsity-inducing prior
• Sticky PDMP samplers for sparse and local inference problems
• Variational Inference in high-dimensional linear regression
• On the Computational Complexity of High-Dimensional Bayesian Variable Selection
• Engagement dynamics and sensitivity analysis of YouTube videos
• Bayesian Adaptive Selection of Basis Functions for Functional Data Representation
• New Development of Bayesian Variable Selection Criteria for Spatial Point Process with Applications
• Adaptive Uncertainty-Guided Model Selection for Data-Driven PDE Discovery
• A Scalable Empirical Bayes Approach to Variable Selection in Generalized Linear Models
• Latent Theme Dictionary Model for Finding Co-occurrent Patterns in Process Data
• Beyond Low Rank: A Data-Adaptive Tensor Completion Method
• A Moreau-Yosida approximation scheme for a class of high-dimensional posterior distributions
• Consistent Sparse Deep Learning: Theory and Computation
• Bayesian Structure Learning for Markov Random Fields with a Spike and Slab Prior
• Robust and Parallel Bayesian Model Selection
• Block Hyper-g Priors in Bayesian Regression
• Latent space projection predictive inference
• Sparse Factor Analysis for Learning and Content Analytics
• VarFA: A Variational Factor Analysis Framework For Efficient Bayesian Learning Analytics
• Bayesian Hypothesis Test using Nonparametric Belief Propagation for Noisy Sparse Recovery
• Empirical Bayes, SURE and Sparse Normal Mean Models
• Geographically Weighted Regression Analysis for Spatial Economics Data: a Bayesian Recourse
• Selective linear segmentation for detecting relevant parameter changes
• Bayesian subset selection and variable importance for interpretable prediction and classification
• Efficient Variational Bayes Learning of Graphical Models with Smooth Structural Changes
• Heterogeneity Pursuit for Spatial Point Pattern with Application to Tree Locations: A Bayesian Semiparametric Recourse
• An approach to large-scale Quasi-Bayesian inference with spike-and-slab priors
• Correlated Random Measures
• On Detection-Directed Estimation Approach for Noisy Compressive Sensing
• Nearly Optimal Variational Inference for High Dimensional Regression with Shrinkage Priors
• Sparse and spurious: dictionary learning with noise and outliers
• Consistent Bayesian Information Criterion Based on a Mixture Prior for Possibly High-Dimensional Multivariate Linear Regression Models
• Bayesian Mixed Effect Sparse Tensor Response Regression Model with Joint Estimation of Activation and Connectivity
• Investigating Competition in Financial Markets: A Sparse Autologistic Model for Dynamic Network Data
• Modeling User Exposure in Recommendation
• Non-convex image reconstruction via Expectation Propagation
• Degree distributions in networks: beyond the power law
• Comment on García-Donato et al. (2025) "Model uncertainty and missing data: An objective Bayesian perspective"
• Invited Discussion of "Model Uncertainty and Missing Data: An Objective Bayesian Perspective" by Gonzalo García-Donato , María Eugenia Castellanos , Stefano Cabras Alicia Quirós , and Anabel Forte
• Learning from a lot: Empirical Bayes in high-dimensional prediction settings
• Bayesian Structure Learning for Markov Random Fields with a Spike and Slab Prior
• Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes
• Covariate-Assisted Bayesian Graph Learning for Heterogeneous Data
• Dynamic Sparse Factor Analysis
• Nonparametric Empirical Bayes Estimation and Testing for Sparse and Heteroscedastic Signals
• P-splines with an l1 penalty for repeated measures
• Mixed Effect Modeling and Variable Selection for Quantile Regression
• Bayesian Variable Selection in Double Generalized Linear Tweedie Spatial Process Models
• Bayesian equation selection on sparse data for discovery of stochastic dynamical systems
• Risk and Regret of Hierarchical Bayesian Learners
• Streaming Probabilistic Deep Tensor Factorization
• Decoupling Shrinkage and Selection for the Bayesian Quantile Regression
• Bayesian Hypothesis Test for Sparse Support Recovery using Belief Propagation
• Detection-Directed Sparse Estimation using Bayesian Hypothesis Test and Belief Propagation
• Nonparametric Causal Feature Selection for Spatiotemporal Risk Mapping of Malaria Incidence in Madagascar
• Analysis of high-dimensional Continuous Time Markov Chains using the Local Bouncy Particle Sampler
• Bayesian Inference for High Dimensional Changing Linear Regression with Application to Minnesota House Price Index Data
• Bayesian Matrix Completion Approach to Causal Inference with Panel Data
• Bayesian Poisson Log-normal Model with Regularized Time Structure for Mortality Projection of Multi-population
• spikeSlabGAM: Bayesian Variable Selection, Model Choice and Regularization for Generalized Additive Mixed Models in R
• Bayesian Analysis of Multiway Tables in Association Studies: A Model Comparison Approach
• Graph Gamma Process Generalized Linear Dynamical Systems
• Sparse Communication via Mixed Distributions
• Bayesian iterative screening in ultra-high dimensional linear regressions
• Robust Bayesian causal estimation for causal inference in medical diagnosis
• Bayesian Inference with the l1-ball Prior: Solving Combinatorial Problems with Exact Zeros
• Approximate spectral gaps for Markov chains mixing times in high dimensions
• Statistical Inference for Bayesian Risk Minimization via Exponentially Tilted Empirical Likelihood
• To Bayes or Not To Bayes? That's no longer the question!
• Bayesian Regularization for Graphical Models with Unequal Shrinkage
• Nonparametric Bayesian Sparse Graph Linear Dynamical Systems
• Adaptive shrinkage of smooth functional effects towards a predefined functional subspace
• Bayesian Model Averaging for Causality Estimation and its Approximation based on Gaussian Scale Mixture Distributions
• Posterior distributions for Hierarchical Spike and Slab Indian Buffet processes
• pivmet: Pivotal Methods for Bayesian Relabelling and k-Means Clustering
• Reconciling the Discrete-Continuous Divide: Towards a Mathematical Theory of Sparse Communication
• Composite mixture of log-linear models for categorical data
• Orthogonalized smoothing for rescaled spike and slab models
• Discriminative models for robust image classification
• Bayesian Changepoint Analysis
• A Scalable Empirical Bayes Approach to Variable Selection
• Bayesian Nonparametric Graph Clustering
• Bayesian-EUCLID: discovering hyperelastic material laws with uncertainties
• A robust Bayesian analysis of variable selection under prior ignorance
• Dictionary Learning Strategies for Compressed Fiber Sensing Using a Probabilistic Sparse Model
• Regularizing Bayesian Predictive Regressions
• Group Spike and Slab Variational Bayes
• Sparse Bayesian Multidimensional Item Response Theory
• Tumor Radiogenomics with Bayesian Layered Variable Selection
• Multilinear Dirichlet Processes
• Semiparametric Functional Factor Models with Bayesian Rank Selection
• Directionally Dependent Multi-View Clustering Using Copula Model
• Covariate-Elaborated Robust Partial Information Transfer with Conditional Spike-and-Slab Prior
• Variable fusion for Bayesian linear regression via spike-and-slab priors
• The ARR2 prior: flexible predictive prior definition for Bayesian auto-regressions
• Disjunct Support Spike and Slab Priors for Variable Selection in Regression under Quasi-sparseness
• Cross-validation and Peeling Strategies for Survival Bump Hunting using Recursive Peeling Methods
• Bayesian Group Selection in Logistic Regression with Application to MRI Data Analysis
• Probabilistic learning of boolean functions applied to the binary classification problem with categorical covariates
• Large-P Variable Selection in Two-Stage Models
• Bayesian Fusion Estimation via t-Shrinkage
• CBA: Contextual Quality Adaptation for Adaptive Bitrate Video Streaming (Extended Version)
• Inducing Sparse Coding and And-Or Grammar from Generator Network
• Multivariate Spatial-Temporal Variable Selection with Applications to Seasonal Tropical Cyclone Modeling
• A Bayesian State-Space Approach to Mapping Directional Brain Networks
• Bayesian Variable Selection for Single Index Logistic Model
• Bayesian Singular Value Regularization via a Cumulative Shrinkage Process
• Algorithmic Bayesian Group Gibbs Selection
• Multiscale Shrinkage and Lévy Processes
• An Approximate Bayesian Approach to Model-assisted Survey Estimation with Many Auxiliary Variables
• Bayesian Variable Selection on Model Spaces Constrained by Heredity Conditions
• Inducing Hierarchical Compositional Model by Sparsifying Generator Network
• A Bayesian Finite Mixture Model with Variable Selection for Data with Mixed-type Variables
• Data-driven discovery of interpretable causal relations for deep learning material laws with uncertainty propagation
• Bayesian Nonparametric Feature and Policy Learning for Decision-Making
• Simultaneous Credible Regions for Multiple Changepoint Locations
• Outcome-guided Bayesian Clustering for Disease Subtype Discovery Using High-dimensional Transcriptomic Data