Robust, randomized preconditioning for kernel ridge regression

Abstract

We investigate preconditioned conjugate gradient methods for kernel ridge regression (KRR) problems with a moderate to large number of data points ($10^4\le N\le 10^7$). We develop and analyze two randomized preconditioners with complementary guarantees. For full-data KRR, RPCholesky preconditioning requires $O(N^2)$ arithmetic operations to achieve fixed accuracy under sufficiently rapid eigenvalue decay of the kernel matrix. For restricted KRR with $k\ll N$ centers, KRILL preconditioning requires $O((N+k^2)k\log k)$ operations with no eigenvalue-decay assumption. Experiments on benchmark and scientific data sets demonstrate the robustness of both methods relative to existing preconditioners.

23 pages, 11 figures