L2 Regularization for Learning Kernels

Abstract

The choice of the kernel is critical to the success of many learning algorithms but it is typically left to the user. Instead, the training data can be used to learn the kernel by selecting it out of a given family, such as that of non-negative linear combinations of p base kernels, constrained by a trace or L1 regularization. This paper studies the problem of learning kernels with the same family of kernels but with an L2 regularization instead, and for regression problems. We analyze the problem of learning kernels with ridge regression. We derive the form of the solution of the optimization problem and give an efficient iterative algorithm for computing that solution. We present a novel theoretical analysis of the problem based on stability and give learning bounds for orthogonal kernels that contain only an additive term O(pp/m) when compared to the standard kernel ridge regression stability bound. We also report the results of experiments indicating that L1 regularization can lead to modest improvements for a small number of kernels, but to performance degradations in larger-scale cases. In contrast, L2 regularization never degrades performance and in fact achieves significant improvements with a large number of kernels.

Appears in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI2009)

References in corpus (1)

• Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning

Cited by in corpus (42)

• Algorithms for Learning Kernels Based on Centered Alignment
• Kernel-based Reconstruction of Graph Signals
• Fast learning rate of multiple kernel learning: Trade-off between sparsity and smoothness
• Uncertainty Aware Learning for High Energy Physics
• A multi-theoretical kernel-based approach to social network-based recommendation
• Online Graph-Adaptive Learning with Scalability and Privacy
• The Local Rademacher Complexity of Lp-Norm Multiple Kernel Learning
• Sparsity-accuracy trade-off in MKL
• Deep Learning for Identifying Iran's Cultural Heritage Buildings in Need of Conservation Using Image Classification and Grad-CAM
• A Unifying View of Multiple Kernel Learning
• A deep learning approach for lower back-pain risk prediction during manual lifting
• Minimizing inter-subject variability in fNIRS based Brain Computer Interfaces via multiple-kernel support vector learning
• Bayesian Efficient Multiple Kernel Learning
• Matrix Element Regression with Deep Neural Networks -- breaking the CPU barrier
• An Active Galactic Nucleus Recognition Model based on Deep Neural Network
• Regularization Strategies and Empirical Bayesian Learning for MKL
• Multiple Kernel Learning from Noisy Labels by Stochastic Programming
• Random Feature-based Online Multi-kernel Learning in Environments with Unknown Dynamics
• Insights from Classifying Visual Concepts with Multiple Kernel Learning
• Multiple Kernel Learning: A Unifying Probabilistic Viewpoint
• Learning the kernel matrix via predictive low-rank approximations
• DFNets: Spectral CNNs for Graphs with Feedback-Looped Filters
• An Online Attention-based Model for Speech Recognition
• Non-Sparse Regularization for Multiple Kernel Learning
• Active Learning with Multiple Kernels
• EPick: Multi-Class Attention-based U-shaped Neural Network for Earthquake Detection and Seismic Phase Picking
• Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks
• Clusternets: A deep learning approach to probe clustering dark energy
• Learning Discriminative Metrics via Generative Models and Kernel Learning
• AI and Holistic Review: Informing Human Reading in College Admissions
• Fast Convergence Rate of Multiple Kernel Learning with Elastic-net Regularization
• Multiple Kernel Learning and Automatic Subspace Relevance Determination for High-dimensional Neuroimaging Data
• A compressive multi-kernel method for privacy-preserving machine learning
• Sparse Multiple Kernel Learning with Geometric Convergence Rate
• Smart Journey in Istanbul: A Mobile Application in Smart Cities for Traffic Estimation by Harnessing Time Series
• ${\ell }_p$-Norm Multiple Kernel One-Class Fisher Null-Space
• Skeptical Deep Learning with Distribution Correction
• A Metric-learning based framework for Support Vector Machines and Multiple Kernel Learning
• Sharp Convergence Rate and Support Consistency of Multiple Kernel Learning with Sparse and Dense Regularization
• Fast Learning Rate of lp-MKL and its Minimax Optimality
• Alignment Based Kernel Learning with a Continuous Set of Base Kernels
• Optimality Implies Kernel Sum Classifiers are Statistically Efficient