Replica Analysis for Ensemble Techniques in Variable Selection

Abstract

Variable selection is a problem of statistics that aims to find the subset of the $N$-dimensional possible explanatory variables that are truly related to the generation process of the response variable. In high-dimensional setups, where the input dimension $N$ is comparable to the data size $M$, it is difficult to use classic methods based on $p$-values. Therefore, methods based on the ensemble learning are often used. In this review article, we introduce how the performance of these ensemble-based methods can be systematically analyzed using the replica method from statistical mechanics when $N$ and $M$ diverge at the same rate as $N,M\to \infty ,M/N\to \alpha \in (0,\infty )$. As a concrete application, we analyze the power of stability selection (SS) and the derandomized knockoff (dKO) with the ${\ell }_1$-regularized statistics in the high-dimensional linear model. The result indicates that dKO provably outperforms the vanilla knockoff and the standard SS, while increasing the bootstrap resampling rate in SS might further improve the detection power.

arXiv comment25 pages, 4 figures, 1 table

References in corpus (13)

• Big Data of Materials Science - Critical Role of the Descriptor
• On asymptotically optimal confidence regions and tests for high-dimensional models
• Controlling the false discovery rate via knockoffs
• Statistical physics of inference: Thresholds and algorithms
• Variable selection with error control: Another look at Stability Selection
• A typical reconstruction limit of compressed sensing based on Lp-norm minimization
• Exhaustive search for sparse variable selection in linear regression
• Variable Selection for Modeling the Absolute Magnitude at Maximum of Type Ia Supernovae
• A statistical mechanics approach to de-biasing and uncertainty estimation in LASSO for random measurements
• Statistical Mechanics of Learning: A Variational Approach for Real Data
• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing
• Semi-analytic approximate stability selection for correlated data in generalized linear models
• Role of Bootstrap Averaging in Generalized Approximate Message Passing

Cited by in corpus (1)

• Variational Gaussian Approximation in Replica Analysis of Parametric Models