Nonparametric sparsity and regularization

Abstract

In this work we are interested in the problems of supervised learning and variable selection when the input-output dependence is described by a nonlinear function depending on a few variables. Our goal is to consider a sparse nonparametric model, hence avoiding linear or additive models. The key idea is to measure the importance of each variable in the model by making use of partial derivatives. Based on this intuition we propose a new notion of nonparametric sparsity and a corresponding least squares regularization scheme. Using concepts and results from the theory of reproducing kernel Hilbert spaces and proximal methods, we show that the proposed learning algorithm corresponds to a minimization problem which can be provably solved by an iterative procedure. The consistency properties of the obtained estimator are studied both in terms of prediction and selection performance. An extensive empirical analysis shows that the proposed method performs favorably with respect to the state-of-the-art methods.

45 pages, 11 figures

References in corpus (20)

• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Consistency of the group Lasso and multiple kernel learning
• The composite absolute penalties family for grouped and hierarchical variable selection
• Component selection and smoothing in multivariate nonparametric regression
• A Unified Framework for High-Dimensional Analysis of M-Estimators with Decomposable Regularizers
• Sparsity in multiple kernel learning
• Accelerating gradient projection methods for ${\ell }_1$-constrained signal recovery by steplength selection rules
• On the performance of algorithms for the minimization of ${\ell }_1$-penalized functionals
• High-Dimensional Non-Linear Variable Selection through Hierarchical Kernel Learning
• Structured Sparsity and Generalization
• Selection of variables and dimension reduction in high-dimensional non-parametric regression

Cited by in corpus (15)

• Optimal Rates for Random Fourier Features
• Variable selection in functional data classification: a maxima-hunting proposal
• Improved spectral convergence rates for graph Laplacians on epsilon-graphs and k-NN graphs
• A Random Block-Coordinate Douglas-Rachford Splitting Method with Low Computational Complexity for Binary Logistic Regression
• Learning Sparse Nonparametric DAGs
• On Kernel Derivative Approximation with Random Fourier Features
• Structured nonlinear variable selection
• A polynomial-time algorithm for learning nonparametric causal graphs
• Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
• A stochastic inertial forward-backward splitting algorithm for multivariate monotone inclusions
• Stochastic inertial primal-dual algorithms
• Dimension Reduction in Contextual Online Learning via Nonparametric Variable Selection
• Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning
• A Proximal Approach for Sparse Multiclass SVM
• Structure learning via unstructured kernel-based M-regression