Convergence rates of Kernel Conjugate Gradient for random design regression

Abstract

We prove statistical rates of convergence for kernel-based least squares regression from i.i.d. data using a conjugate gradient algorithm, where regularization against overfitting is obtained by early stopping. This method is related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. Following the setting introduced in earlier related literature, we study so-called "fast convergence rates" depending on the regularity of the target regression function (measured by a source condition in terms of the kernel integral operator) and on the effective dimensionality of the data mapped into the kernel space. We obtain upper bounds, essentially matching known minimax lower bounds, for the ${\mathcal{L}}^2$ (prediction) norm as well as for the stronger Hilbert norm, if the true regression function belongs to the reproducing kernel Hilbert space. If the latter assumption is not fulfilled, we obtain similar convergence rates for appropriate norms, provided additional unlabeled data are available.

References in corpus (1)

• Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii

Cited by in corpus (6)

• Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition
• Kernel Conjugate Gradient Methods with Random Projections
• Kernel Interpolation of High Dimensional Scattered Data
• Early stopping and polynomial smoothing in regression with reproducing kernels
• Adaptive Stopping Rule for Kernel-based Gradient Descent Algorithms
• Analysis of regularized Nyström subsampling for regression functions of low smoothness