Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning

Abstract

For supervised and unsupervised learning, positive definite kernels allow to use large and potentially infinite dimensional feature spaces with a computational cost that only depends on the number of observations. This is usually done through the penalization of predictor functions by Euclidean or Hilbertian norms. In this paper, we explore penalizing by sparsity-inducing norms such as the l1-norm or the block l1-norm. We assume that the kernel decomposes into a large sum of individual basis kernels which can be embedded in a directed acyclic graph; we show that it is then possible to perform kernel selection through a hierarchical multiple kernel learning framework, in polynomial time in the number of selected kernels. This framework is naturally applied to non linear variable selection; our extensive simulations on synthetic datasets and datasets from the UCI repository show that efficiently exploring the large feature space through sparsity-inducing norms leads to state-of-the-art predictive performance.

References in corpus (1)

• Consistency of the group Lasso and multiple kernel learning

Cited by in corpus (32)

• Structured Variable Selection with Sparsity-Inducing Norms
• Algorithms for Learning Kernels Based on Centered Alignment
• Structure Discovery in Nonparametric Regression through Compositional Kernel Search
• L2 Regularization for Learning Kernels
• Proximal Methods for Hierarchical Sparse Coding
• Smoothing proximal gradient method for general structured sparse regression
• Group Lasso with Overlaps: the Latent Group Lasso approach
• Network Flow Algorithms for Structured Sparsity
• Fast learning rate of multiple kernel learning: Trade-off between sparsity and smoothness
• Structured sparsity-inducing norms through submodular functions
• High-Dimensional Non-Linear Variable Selection through Hierarchical Kernel Learning
• Hierarchical Sparse Modeling: A Choice of Two Group Lasso Formulations
• Differentiable Compositional Kernel Learning for Gaussian Processes
• Supervised Feature Selection in Graphs with Path Coding Penalties and Network Flows
• Model-Consistent Sparse Estimation through the Bootstrap
• Positive Definite Kernels in Machine Learning
• Multi-View Spectral Clustering with High-Order Optimal Neighborhood Laplacian Matrix
• Accelerated iterative regularization via dual diagonal descent
• A Randomized Mirror Descent Algorithm for Large Scale Multiple Kernel Learning
• Ultra High-Dimensional Nonlinear Feature Selection for Big Biological Data
• New Generalization Bounds for Learning Kernels
• A Convex Feature Learning Formulation for Latent Task Structure Discovery
• CRH: A Simple Benchmark Approach to Continuous Hashing
• Ensembles of Kernel Predictors
• Learning Hierarchical Interactions at Scale: A Convex Optimization Approach
• On the Expressive Power of Kernel Methods and the Efficiency of Kernel Learning by Association Schemes
• String Gaussian Process Kernels
• Learning Data-adaptive Nonparametric Kernels
• A compressive multi-kernel method for privacy-preserving machine learning
• Sparse Multiple Kernel Learning with Geometric Convergence Rate
• Learning Explicit Deep Representations from Deep Kernel Networks
• Voted Kernel Regularization