Scaling Gaussian Process Regression with Derivatives

Abstract

Gaussian processes (GPs) with derivatives are useful in many applications, including Bayesian optimization, implicit surface reconstruction, and terrain reconstruction. Fitting a GP to function values and derivatives at $n$ points in $d$ dimensions requires linear solves and log determinants with an $n(d+1)\times n(d+1)$ positive definite matrix -- leading to prohibitive $\mathcal{O}(n^3{d}^3)$ computations for standard direct methods. We propose iterative solvers using fast $\mathcal{O}(nd)$ matrix-vector multiplications (MVMs), together with pivoted Cholesky preconditioning that cuts the iterations to convergence by several orders of magnitude, allowing for fast kernel learning and prediction. Our approaches, together with dimensionality reduction, enables Bayesian optimization with derivatives to scale to high-dimensional problems and large evaluation budgets.

Appears at Advances in Neural Information Processing Systems 32 (NIPS), 2018

References in corpus (3)

• Bayesian Optimization with Gradients
• Product Kernel Interpolation for Scalable Gaussian Processes
• Scalable Log Determinants for Gaussian Process Kernel Learning

Cited by in corpus (12)

• Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization
• Basic Ideas and Tools for Projection-Based Model Reduction of Parametric Partial Differential Equations
• Symplectic Gaussian Process Regression of Hamiltonian Flow Maps
• Scalable Constrained Bayesian Optimization
• Efficient force field and energy emulation through partition of permutationally equivalent atoms
• Trading Convergence Rate with Computational Budget in High Dimensional Bayesian Optimization
• Using Distance Correlation for Efficient Bayesian Optimization
• Hierarchical Gaussian Processes with Wasserstein-2 Kernels
• Kernel Conditional Density Operators
• Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning
• Large-scale magnetic field maps using structured kernel interpolation for Gaussian process regression
• GPGM-SLAM: a Robust SLAM System for Unstructured Planetary Environments with Gaussian Process Gradient Maps