Consistency and robustness of kernel-based regression in convex risk minimization

Abstract

We investigate statistical properties for a broad class of modern kernel-based regression (KBR) methods. These kernel methods were developed during the last decade and are inspired by convex risk minimization in infinite-dimensional Hilbert spaces. One leading example is support vector regression. We first describe the relationship between the loss function $L$ of the KBR method and the tail of the response variable. We then establish the $L$-risk consistency for KBR which gives the mathematical justification for the statement that these methods are able to ``learn''. Then we consider robustness properties of such kernel methods. In particular, our results allow us to choose the loss function and the kernel to obtain computationally tractable and consistent KBR methods that have bounded influence functions. Furthermore, bounds for the bias and for the sensitivity curve, which is a finite sample version of the influence function, are developed, and the relationship between KBR and classical $M$ estimators is discussed.

Published at http://dx.doi.org/10.3150/07-BEJ5102 in the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

Cited by in corpus (24)

• Robustness and Regularization of Support Vector Machines
• Distributionally Robust Optimization: A Review
• Robust Kernel Density Estimation
• Multi-View Matrix Completion for Multi-Label Image Classification
• Convolutional Gaussian Processes
• Privacy-preserving parametric inference: a case for robust statistics
• Consistency of support vector machines for forecasting the evolution of an unknown ergodic dynamical system from observations with unknown noise
• Robust Kernel-based Distribution Regression
• Convergence Guarantees for Gaussian Process Means With Misspecified Likelihoods and Smoothness
• Kernel Distributionally Robust Optimization
• A Statistical Learning Assessment of Huber Regression
• Qualitative Robustness of Support Vector Machines
• Robust Kernel (Cross-) Covariance Operators in Reproducing Kernel Hilbert Space toward Kernel Methods
• Total Stability of SVMs and Localized SVMs
• Quantifying Epistemic Uncertainty in Deep Learning
• Doubly Robust Stein-Kernelized Monte Carlo Estimator: Simultaneous Bias-Variance Reduction and Supercanonical Convergence
• Identifying Outliers using Influence Function of Multiple Kernel Canonical Correlation Analysis
• Asymptotic Normality of Support Vector Machine Variants and Other Regularized Kernel Methods
• Model-specific Data Subsampling with Influence Functions
• How rotational invariance of common kernels prevents generalization in high dimensions
• Gene-Gene association for Imaging Genetics Data using Robust Kernel Canonical Correlation Analysis
• Prediction of Dynamical time Series Using Kernel Based Regression and Smooth Splines
• Practical Tikhonov Regularized Estimators in Reproducing Kernel Hilbert Spaces for Statistical Inverse Problems
• Kernel-based Partial Permutation Test for Detecting Heterogeneous Functional Relationship