Tight conditions for consistency of variable selection in the context of high dimensionality

Abstract

We address the issue of variable selection in the regression model with very high ambient dimension, that is, when the number of variables is very large. The main focus is on the situation where the number of relevant variables, called intrinsic dimension, is much smaller than the ambient dimension d. Without assuming any parametric form of the underlying regression function, we get tight conditions making it possible to consistently estimate the set of relevant variables. These conditions relate the intrinsic dimension to the ambient dimension and to the sample size. The procedure that is provably consistent under these tight conditions is based on comparing quadratic functionals of the empirical Fourier coefficients with appropriately chosen threshold values. The asymptotic analysis reveals the presence of two quite different re gimes. The first regime is when the intrinsic dimension is fixed. In this case the situation in nonparametric regression is the same as in linear regression, that is, consistent variable selection is possible if and only if log d is small compared to the sample size n. The picture is different in the second regime, that is, when the number of relevant variables denoted by s tends to infinity as $n\to \infty$. Then we prove that consistent variable selection in nonparametric set-up is possible only if s+loglog d is small compared to log n. We apply these results to derive minimax separation rates for the problem of variable

arXiv admin note: text overlap with arXiv:1102.3616; Published in at http://dx.doi.org/10.1214/12-AOS1046 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (12)

• Nearly unbiased variable selection under minimax concave penalty
• The composite absolute penalties family for grouped and hierarchical variable selection
• Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem
• High-dimensional Ising model selection using ${\ell }_1$-regularized logistic regression
• Structured Variable Selection with Sparsity-Inducing Norms
• High-dimensional variable selection
• Sparsity in multiple kernel learning
• Rodeo: Sparse, greedy nonparametric regression
• High-Dimensional Non-Linear Variable Selection through Hierarchical Kernel Learning
• Selection of variables and dimension reduction in high-dimensional non-parametric regression
• Optimal adaptive estimation of a quadratic functional
• Tight conditions for consistent variable selection in high dimensional nonparametric regression

Cited by in corpus (5)

• Minimax-optimal nonparametric regression in high dimensions
• All-In-One Robust Estimator of the Gaussian Mean
• Randomized maximum-contrast selection: subagging for large-scale regression
• Variable Selection with ABC Bayesian Forests
• Optimal detection of the feature matching map in presence of noise and outliers