Parametric Gaussian Process Regressors
arXiv:1910.07123
Abstract
The combination of inducing point methods with stochastic variational inference has enabled approximate Gaussian Process (GP) inference on large datasets. Unfortunately, the resulting predictive distributions often exhibit substantially underestimated uncertainties. Notably, in the regression case the predictive variance is typically dominated by observation noise, yielding uncertainty estimates that make little use of the input-dependent function uncertainty that makes GP priors attractive. In this work we propose two simple methods for scalable GP regression that address this issue and thus yield substantially improved predictive uncertainties. The first applies variational inference to FITC (Fully Independent Training Conditional; Snelson et.~al.~2006). The second bypasses posterior approximations and instead directly targets the posterior predictive distribution. In an extensive empirical comparison with a number of alternative methods for scalable GP regression, we find that the resulting predictive distributions exhibit significantly better calibrated uncertainties and higher log likelihoods--often by as much as half a nat per datapoint.
arXiv comment17 pages, 10 figures; as appeared in ICML 2020
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
- Scalable Variational Gaussian Process Classification
- Dropout Inference in Bayesian Neural Networks with Alpha-divergences
- When Gaussian Process Meets Big Data: A Review of Scalable GPs
- Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
- Generalized Variational Inference: Three arguments for deriving new Posteriors
- MCMC for Variationally Sparse Gaussian Processes
- Exact Gaussian Processes on a Million Data Points
- Orthogonally Decoupled Variational Gaussian Processes
- Robust Deep Gaussian Processes
- Monte Carlo Structured SVI for Two-Level Non-Conjugate Models