Selection of variables and dimension reduction in high-dimensional non-parametric regression

Abstract

We consider a $l_1$-penalization procedure in the non-parametric Gaussian regression model. In many concrete examples, the dimension $d$ of the input variable $X$ is very large (sometimes depending on the number of observations). Estimation of a $\beta$-regular regression function $f$ cannot be faster than the slow rate $n^{-2\beta /(2\beta +d)}$. Hopefully, in some situations, $f$ depends only on a few numbers of the coordinates of $X$. In this paper, we construct two procedures. The first one selects, with high probability, these coordinates. Then, using this subset selection method, we run a local polynomial estimator (on the set of interesting coordinates) to estimate the regression function at the rate $n^{-2\beta /(2\beta +d^{\ast })}$, where $d^{\ast }$, the "real" dimension of the problem (exact number of variables whom $f$ depends on), has replaced the dimension $d$ of the design. To achieve this result, we used a $l_1$ penalization method in this non-parametric setup.

Published in at http://dx.doi.org/10.1214/08-EJS327 the Electronic Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (6)

• Simultaneous analysis of Lasso and Dantzig selector
• Lasso-type recovery of sparse representations for high-dimensional data
• Fast learning rates for plug-in classifiers
• Local polynomial regression on unknown manifolds
• Rodeo: Sparse, greedy nonparametric regression
• Fast learning rates for plug-in classifiers under the margin condition

Cited by in corpus (9)

• Nonparametric sparsity and regularization
• Tight conditions for consistency of variable selection in the context of high dimensionality
• Learning Sparse Nonparametric DAGs
• Tight conditions for consistent variable selection in high dimensional nonparametric regression
• Adaptive Estimation of Multivariate Piecewise Polynomials and Bounded Variation Functions by Optimal Decision Trees
• Additive Gaussian Process Regression
• A polynomial-time algorithm for learning nonparametric causal graphs
• Dimension Reduction in Contextual Online Learning via Nonparametric Variable Selection
• Variable Selection Consistency of Gaussian Process Regression