Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes

Abstract

Variable selection in Gaussian processes (GPs) is typically undertaken by thresholding the inverse lengthscales of automatic relevance determination kernels, but in high-dimensional datasets this approach can be unreliable. A more probabilistically principled alternative is to use spike and slab priors and infer a posterior probability of variable inclusion. However, existing implementations in GPs are very costly to run in both high-dimensional and large-$n$ datasets, or are only suitable for unsupervised settings with specific kernels. As such, we develop a fast and scalable variational inference algorithm for the spike and slab GP that is tractable with arbitrary differentiable kernels. We improve our algorithm's ability to adapt to the sparsity of relevant variables by Bayesian model averaging over hyperparameters, and achieve substantial speed ups using zero temperature posterior restrictions, dropout pruning and nearest neighbour minibatching. In experiments our method consistently outperforms vanilla and sparse variational GPs whilst retaining similar runtimes (even when $n=10^6$) and performs competitively with a spike and slab GP using MCMC but runs up to $1000$ times faster.

arXiv commentAccepted at the 25th International Conference on Artificial Intelligence and Statistics (AISTATS 2022)

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Nearly unbiased variable selection under minimax concave penalty
• Spike and slab variable selection: Frequentist and Bayesian strategies
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Bayesian Compression for Deep Learning
• Bayesian variable selection with shrinking and diffusing priors
• Efficient leave-one-out cross-validation for Bayesian non-factorized normal and Student-t models
• Variable selection for Gaussian processes via sensitivity analysis of the posterior predictive distribution
• On Using Control Variates with Stochastic Approximation for Variational Bayes and its Connection to Stochastic Linear Regression
• Rao-Blackwellized Stochastic Gradients for Discrete Distributions
• Additive Gaussian Process Regression
• Spike and Slab Gaussian Process Latent Variable Models
• Scalable Cross Validation Losses for Gaussian Process Models