When Gaussian Process Meets Big Data: A Review of Scalable GPs

Abstract

The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process (GP) regression, a well-known non-parametric and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. But they have not yet been comprehensively reviewed and analyzed in order to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this paper is devoted to the review on state-of-the-art scalable GPs involving two main categories: global approximations which distillate the entire data and local approximations which divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations which modify the prior but perform exact inference, posterior approximations which retain exact prior but perform approximate inference, and structured sparse approximations which exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues regarding the implementation of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.

arXiv comment20 pages, 6 figures

References in corpus (49)

• Adam: A Method for Stochastic Optimization
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Gaussian Processes for Big Data
• Scalable Variational Gaussian Process Classification
• Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
• Variational Gaussian Process State-Space Models
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Bayesian Hierarchical Mixtures of Experts
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• GPz: Non-stationary sparse Gaussian processes for heteroscedastic uncertainty estimation in photometric redshifts
• Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions
• Stochastic Variational Deep Kernel Learning
• Random Feature Expansions for Deep Gaussian Processes
• Bayesian Inference and Learning in Gaussian Process State-Space Models with Particle MCMC
• Fast Variational Inference in the Conjugate Exponential Family
• A Sparse Gaussian Process Framework for Photometric Redshift Estimation
• Thoughts on Massively Scalable Gaussian Processes
• Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models
• Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
• Variational Auto-encoded Deep Gaussian Processes
• Decentralized Data Fusion and Active Sensing with Mobile Sensors for Modeling and Predicting Spatiotemporal Traffic Phenomena
• MCMC for Variationally Sparse Gaussian Processes
• Nested Kriging predictions for datasets with large number of observations
• Product Kernel Interpolation for Scalable Gaussian Processes
• Improving Output Uncertainty Estimation and Generalization in Deep Learning via Neural Network Gaussian Processes
• Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification
• Exact Gaussian Processes on a Million Data Points
• Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs
• Multiresolution Gaussian Processes
• Recurrent Gaussian Processes
• Gaussian Process Models with Parallelization and GPU acceleration
• Modelling local and global phenomena with sparse Gaussian processes
• Parallel Gaussian Process Regression with Low-Rank Covariance Matrix Approximations
• Variational Inference for Uncertainty on the Inputs of Gaussian Process Models
• Patchwork Kriging for Large-scale Gaussian Process Regression
• An asymptotic analysis of distributed nonparametric methods
• Deep Gaussian Processes with Convolutional Kernels
• Preconditioning Kernel Matrices
• Variational inference for sparse spectrum Gaussian process regression
• A Short Note on Gaussian Process Modeling for Large Datasets using Graphics Processing Units
• Deep Gaussian Covariance Network
• Blitzkriging: Kronecker-structured Stochastic Gaussian Processes
• On Random Subsampling of Gaussian Process Regression: A Graphon-Based Analysis
• Scalable Gaussian Processes with Billions of Inducing Inputs via Tensor Train Decomposition
• Large-scale Heteroscedastic Regression via Gaussian Process
• Efficient Multiscale Gaussian Process Regression using Hierarchical Clustering
• Scalable Multi-Class Gaussian Process Classification using Expectation Propagation
• Stochastic Variational Inference for Bayesian Sparse Gaussian Process Regression
• EigenGP: Gaussian Process Models with Adaptive Eigenfunctions

Cited by in corpus (31)

• Fast and Accurate Uncertainty Estimation in Chemical Machine Learning
• Reconstructing QCD Spectral Functions with Gaussian Processes
• Exact Gaussian Processes on a Million Data Points
• GPdoemd: a Python package for design of experiments for model discrimination
• Accelerating Bayesian microseismic event location with deep learning
• A machine learning approach for efficient multi-dimensional integration
• Deep Structured Mixtures of Gaussian Processes
• A visual big data system for the prediction of weather-related variables: Jordan-Spain case study
• Sparse Additive Gaussian Process Regression
• Machine learning at the atomic-scale
• Deep Bayesian Gaussian Processes for Uncertainty Estimation in Electronic Health Records
• Towards fast machine-learning-assisted Bayesian posterior inference of microseismic event location and source mechanism
• Parametric Gaussian Process Regressors
• Gaussian Process Optimization with Adaptive Sketching: Scalable and No Regret
• Fast increased fidelity approximate Gibbs samplers for Bayesian Gaussian process regression
• Calibrated simplex-mapping classification
• Recursive Estimation for Sparse Gaussian Process Regression
• Stein Variational Gaussian Processes
• Towards Scalable Gaussian Process Modeling
• Scalable Gaussian Process Classification with Additive Noise for Various Likelihoods
• Scalable Gaussian Process Regression for Kernels with a Non-Stationary Phase
• The Renyi Gaussian Process: Towards Improved Generalization
• Eat, Sleep, Code, Repeat: Tips for Early-Career Researchers in Computational Science
• Statistical Modelling and Analysis of the Computer-Simulated Datasets
• Bayesian Inference Gaussian Process Multiproxy Alignment of Continuous Signals (BIGMACS): Applications for Paleoceanography
• A memory-free spatial additive mixed modeling for big spatial data
• Streamlined Empirical Bayes Fitting of Linear Mixed Models in Mobile Health
• Dynamic Security Assessment of Small-Signal Stability for Power Systems using Windowed Online Gaussian Process
• Planning from Images with Deep Latent Gaussian Process Dynamics
• Orthogonally Decoupled Variational Fourier Features
• Scalable Bayesian Optimization with Sparse Gaussian Process Models