Variational Fourier features for Gaussian processes

Abstract

This work brings together two powerful concepts in Gaussian processes: the variational approach to sparse approximation and the spectral representation of Gaussian processes. This gives rise to an approximation that inherits the benefits of the variational approach but with the representational power and computational scalability of spectral representations. The work hinges on a key result that there exist spectral features related to a finite domain of the Gaussian process which exhibit almost-independent covariances. We derive these expressions for Matern kernels in one dimension, and generalize to more dimensions using kernels with specific structures. Under the assumption of additive Gaussian noise, our method requires only a single pass through the dataset, making for very fast and accurate computation. We fit a model to 4 million training points in just a few minutes on a standard laptop. With non-conjugate likelihoods, our MCMC scheme reduces the cost of computation from O(NM2) (for a sparse Gaussian process) to O(NM) per iteration, where N is the number of data and M is the number of features.

References in corpus (3)

• MCMC for Variationally Sparse Gaussian Processes
• A la Carte - Learning Fast Kernels
• Regularizing Solutions to the MEG Inverse Problem Using Space-Time Separable Covariance Functions

Cited by in corpus (44)

• A Framework for Interdomain and Multioutput Gaussian Processes
• Rates of Convergence for Sparse Variational Gaussian Process Regression
• The Functional Neural Process
• Kernel methods through the roof: handling billions of points efficiently
• Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming
• GPdoemd: a Python package for design of experiments for model discrimination
• Convergence of Sparse Variational Inference in Gaussian Processes Regression
• Scalable Exact Inference in Multi-Output Gaussian Processes
• State Space Gaussian Processes with Non-Gaussian Likelihood
• Efficiently Sampling Functions from Gaussian Process Posteriors
• Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimization
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• Nonnegative spatial factorization
• Stationary Activations for Uncertainty Calibration in Deep Learning
• Probabilistic selection of inducing points in sparse Gaussian processes
• Large-scale Heteroscedastic Regression via Gaussian Process
• Sparse Orthogonal Variational Inference for Gaussian Processes
• Variational Orthogonal Features
• Compositional uncertainty in deep Gaussian processes
• Scalable Training of Inference Networks for Gaussian-Process Models
• Spatio-Temporal Variational Gaussian Processes
• Pathwise Conditioning of Gaussian Processes
• Scalable Thompson Sampling using Sparse Gaussian Process Models
• A Bayes-Sard Cubature Method
• Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach
• SigGPDE: Scaling Sparse Gaussian Processes on Sequential Data
• Recent Advances in Data-Driven Wireless Communication Using Gaussian Processes: A Comprehensive Survey
• Matérn Gaussian Processes on Graphs
• Sparse Spectrum Gaussian Process for Bayesian Optimization
• Inter-domain Deep Gaussian Processes
• SLEIPNIR: Deterministic and Provably Accurate Feature Expansion for Gaussian Process Regression with Derivatives
• Deep Neural Networks as Point Estimates for Deep Gaussian Processes
• End-to-End Probabilistic Inference for Nonstationary Audio Analysis
• Marginalising over Stationary Kernels with Bayesian Quadrature
• Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition
• Adaptive Inducing Points Selection For Gaussian Processes
• Variational Gaussian Processes: A Functional Analysis View
• Latent variable modeling with random features
• Sparse Algorithms for Markovian Gaussian Processes
• Orthogonally Decoupled Variational Fourier Features
• Large-Scale Learning with Fourier Features and Tensor Decompositions
• Spectrum Gaussian Processes Based On Tunable Basis Functions
• Scalable Grouped Gaussian Processes via Direct Cholesky Functional Representations
• The Statistical Cost of Robust Kernel Hyperparameter Tuning