Stein Variational Gaussian Processes
arXiv:2009.12141
Abstract
We show how to use Stein variational gradient descent (SVGD) to carry out inference in Gaussian process (GP) models with non-Gaussian likelihoods and large data volumes. Markov chain Monte Carlo (MCMC) is extremely computationally intensive for these situations, but the parametric assumptions required for efficient variational inference (VI) result in incorrect inference when they encounter the multi-modal posterior distributions that are common for such models. SVGD provides a non-parametric alternative to variational inference which is substantially faster than MCMC. We prove that for GP models with Lipschitz gradients the SVGD algorithm monotonically decreases the Kullback-Leibler divergence from the sampling distribution to the true posterior. Our method is demonstrated on benchmark problems in both regression and classification, a multimodal posterior, and an air quality example with 550,134 spatiotemporal observations, showing substantial performance improvements over MCMC and VI.
arXiv comment26 pages, 5 figures
References in corpus (17)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Gaussian Processes for Big Data
- Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification
- Normalizing Flows for Probabilistic Modeling and Inference
- Elliptical slice sampling
- Stein Variational Gradient Descent as Gradient Flow
- Measuring Sample Quality with Stein's Method
- When Gaussian Process Meets Big Data: A Review of Scalable GPs
- Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
- MCMC for Variationally Sparse Gaussian Processes
- On the geometry of Stein variational gradient descent
- Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
- A Non-Asymptotic Analysis for Stein Variational Gradient Descent
- VAE Learning via Stein Variational Gradient Descent
- Convergence of Sparse Variational Inference in Gaussian Processes Regression
- A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation
- The equivalence between Stein variational gradient descent and black-box variational inference