Projective Inference in High-dimensional Problems: Prediction and Feature Selection

Abstract

This paper discusses predictive inference and feature selection for generalized linear models with scarce but high-dimensional data. We argue that in many cases one can benefit from a decision theoretically justified two-stage approach: first, construct a possibly non-sparse model that predicts well, and then find a minimal subset of features that characterize the predictions. The model built in the first step is referred to as the \emph{reference model} and the operation during the latter step as predictive \emph{projection}. The key characteristic of this approach is that it finds an excellent tradeoff between sparsity and predictive accuracy, and the gain comes from utilizing all available information including prior and that coming from the left out features. We review several methods that follow this principle and provide novel methodological contributions. We present a new projection technique that unifies two existing techniques and is both accurate and fast to compute. We also propose a way of evaluating the feature selection process using fast leave-one-out cross-validation that allows for easy and intuitive model size selection. Furthermore, we prove a theorem that helps to understand the conditions under which the projective approach could be beneficial. The benefits are illustrated via several simulated and real world examples.

References in corpus (29)

• Distilling the Knowledge in a Neural Network
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Visualization in Bayesian workflow
• Spike and slab variable selection: Frequentist and Bayesian strategies
• Optimal predictive model selection
• Using stacking to average Bayesian predictive distributions
• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
• Comparison of Bayesian predictive methods for model selection
• Bayesian variable selection with shrinking and diffusing priors
• Needles and Straw in a Haystack: Posterior concentration for possibly sparse sequences
• Pareto Smoothed Importance Sampling
• "Pre-conditioning" for feature selection and regression in high-dimensional problems
• The Horseshoe Estimator: Posterior Concentration around Nearly Black Vectors
• Generalized Beta Mixtures of Gaussians
• Implicitly Adaptive Importance Sampling
• Projection predictive model selection for Gaussian processes
• Scalable Importance Tempering and Bayesian Variable Selection
• Local Interpretable Model-agnostic Explanations of Bayesian Predictive Models via Kullback-Leibler Projections

Cited by in corpus (19)

• Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data
• Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality
• Uncertainty Quantification and Propagation in Surrogate-based Bayesian Inference
• Intuitive Joint Priors for Bayesian Linear Multilevel Models: The R2D2M2 prior
• Limitations of "Limitations of Bayesian leave-one-out cross-validation for model selection"
• Projection Predictive Inference for Generalized Linear and Additive Multilevel Models
• Predictive Bayesian selection of multistep Markov chains, applied to the detection of the hot hand and other statistical dependencies in free throws
• A fully Bayesian sparse polynomial chaos expansion approach with joint priors on the coefficients and global selection of terms
• Fast, Optimal, and Targeted Predictions using Parametrized Decision Analysis
• Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison
• Efficient estimation and correction of selection-induced bias with order statistics
• Latent space projection predictive inference
• Monotone function estimation in the presence of extreme data coarsening: Analysis of preeclampsia and birth weight in urban Uganda
• Projection predictive variable selection for discrete response families with finite support
• Generalized Decomposition Priors on R2
• On Reparameterization Invariant Bayesian Point Estimates and Credible Regions
• The ARR2 prior: flexible predictive prior definition for Bayesian auto-regressions
• Bayesian clustering using random effects models and predictive projections
• Bayesian model selection in the $\mathcal{M}$-open setting -- Approximate posterior inference and probability-proportional-to-size subsampling for efficient large-scale leave-one-out cross-validation