Exact Gaussian Processes on a Million Data Points

Abstract

Gaussian processes (GPs) are flexible non-parametric models, with a capacity that grows with the available data. However, computational constraints with standard inference procedures have limited exact GPs to problems with fewer than about ten thousand training points, necessitating approximations for larger datasets. In this paper, we develop a scalable approach for exact GPs that leverages multi-GPU parallelization and methods like linear conjugate gradients, accessing the kernel matrix only through matrix multiplication. By partitioning and distributing kernel matrix multiplies, we demonstrate that an exact GP can be trained on over a million points, a task previously thought to be impossible with current computing hardware, in less than 2 hours. Moreover, our approach is generally applicable, without constraints to grid data or specific kernel classes. Enabled by this scalability, we perform the first-ever comparison of exact GPs against scalable GP approximations on datasets with $10^4-10^6$ data points, showing dramatic performance improvements.

arXiv commentPublished at NeurIPS 2019

References in corpus (16)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Gaussian Processes for Big Data
• Scalable Variational Gaussian Process Classification
• Elliptical slice sampling
• Gaussian Process Kernels for Pattern Discovery and Extrapolation
• Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
• Gaussian Process Regression Networks
• Stochastic Variational Deep Kernel Learning
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• Product Kernel Interpolation for Scalable Gaussian Processes
• Orthogonally Decoupled Variational Gaussian Processes
• Preconditioning Kernel Matrices
• Scalable Gaussian Processes with Grid-Structured Eigenfunctions (GP-GRIEF)
• Scalable Gaussian Processes with Billions of Inducing Inputs via Tensor Train Decomposition

Cited by in corpus (11)

• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Time series forecasting with Gaussian Processes needs priors
• Density estimation with Gaussian processes for gravitational-wave posteriors
• Machine Learning with bond information for local structure optimizations in surface science
• Parametric Gaussian Process Regressors
• Sparse Orthogonal Variational Inference for Gaussian Processes
• Fast and Accurate Gaussian Kernel Ridge Regression Using Matrix Decompositions for Preconditioning
• Scalable Grouped Gaussian Processes via Direct Cholesky Functional Representations
• Modulating Scalable Gaussian Processes for Expressive Statistical Learning
• Fenton-Wilkinson Order Statistics and German Tanks: A Case Study of an Orienteering Relay Race