"Pre-conditioning" for feature selection and regression in high-dimensional problems

Abstract

We consider regression problems where the number of predictors greatly exceeds the number of observations. We propose a method for variable selection that first estimates the regression function, yielding a "pre-conditioned" response variable. The primary method used for this initial regression is supervised principal components. Then we apply a standard procedure such as forward stepwise selection or the LASSO to the pre-conditioned response variable. In a number of simulated and real data examples, this two-step procedure outperforms forward stepwise selection or the usual LASSO (applied directly to the raw outcome). We also show that under a certain Gaussian latent variable model, application of the LASSO to the pre-conditioned response variable is consistent as the number of predictors and observations increases. Moreover, when the observational noise is rather large, the suggested procedure can give a more accurate estimate than LASSO. We illustrate our method on some real problems, including survival analysis with microarray data.

References in corpus (1)

• High-dimensional graphs and variable selection with the Lasso

Cited by in corpus (22)

• Robust rank correlation based screening
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Ultrahigh dimensional variable selection: beyond the linear model
• Genomics models in radiotherapy: from mechanistic to machine learning
• Noisy Matrix Completion: Understanding Statistical Guarantees for Convex Relaxation via Nonconvex Optimization
• Preconditioning to comply with the Irrepresentable Condition
• Spectral Deconfounding via Perturbed Sparse Linear Models
• Sure Independence Screening for Ultra-High Dimensional Feature Space
• Robust high dimensional factor models with applications to statistical machine learning
• Identification of significant features in DNA microarray data
• Two-stage Sampling, Prediction and Adaptive Regression via Correlation Screening (SPARCS)
• Predicting phenotypes from microarrays using amplified, initially marginal, eigenvector regression
• A data-dependent weighted LASSO under Poisson noise
• A note relating ridge regression and OLS p-values to preconditioned sparse penalized regression
• Bayesian projection approaches to variable selection and exploring model uncertainty
• Compressed and Penalized Linear Regression
• Predictive modelling of football injuries
• Pre-processing with Orthogonal Decompositions for High-dimensional Explanatory Variables
• Statistical and knowledge supported visualization of multivariate data
• Are we far from correctly inferring gene interaction networks with Lasso?
• A Greedy Homotopy Method for Regression with Nonconvex Constraints
• Sufficient principal component regression for pattern discovery in transcriptomic data