High-Dimensional Non-Linear Variable Selection through Hierarchical Kernel Learning

Abstract

We consider the problem of high-dimensional non-linear variable selection for supervised learning. Our approach is based on performing linear selection among exponentially many appropriately defined positive definite kernels that characterize non-linear interactions between the original variables. To select efficiently from these many kernels, we use the natural hierarchical structure of the problem to extend the multiple kernel learning framework to kernels that can be embedded in a directed acyclic graph; we show that it is then possible to perform kernel selection through a graph-adapted sparsity-inducing norm, in polynomial time in the number of selected kernels. Moreover, we study the consistency of variable selection in high-dimensional settings, showing that under certain assumptions, our regularization framework allows a number of irrelevant variables which is exponential in the number of observations. Our simulations on synthetic datasets and datasets from the UCI repository show state-of-the-art predictive performance for non-linear regression problems.

References in corpus (15)

• Rejoinder to "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Simultaneous analysis of Lasso and Dantzig selector
• Consistency of the group Lasso and multiple kernel learning
• Component selection and smoothing in multivariate nonparametric regression
• Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning
• Testing for Homogeneity with Kernel Fisher Discriminant Analysis

Cited by in corpus (17)

• Structured Variable Selection with Sparsity-Inducing Norms
• Additive Gaussian Processes
• Convex and Network Flow Optimization for Structured Sparsity
• Nonparametric sparsity and regularization
• Tight conditions for consistency of variable selection in the context of high dimensionality
• Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
• Multiresolution Gaussian Processes
• Tight conditions for consistent variable selection in high dimensional nonparametric regression
• Learning Nonlinear Functions Using Regularized Greedy Forest
• Randomly Projected Additive Gaussian Processes for Regression
• A Convex Feature Learning Formulation for Latent Task Structure Discovery
• Structured nonlinear variable selection
• Invariances of random fields paths, with applications in Gaussian Process Regression
• Multi-scale Mining of fMRI data with Hierarchical Structured Sparsity
• Towards Ultrahigh Dimensional Feature Selection for Big Data
• Kernel Identification Through Transformers
• The Minecraft Kernel: Modelling correlated Gaussian Processes in the Fourier domain