Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Abstract

Gaussian processes (GPs) are a powerful tool for probabilistic inference over functions. They have been applied to both regression and non-linear dimensionality reduction, and offer desirable properties such as uncertainty estimates, robustness to over-fitting, and principled ways for tuning hyper-parameters. However the scalability of these models to big datasets remains an active topic of research. We introduce a novel re-parametrisation of variational inference for sparse GP regression and latent variable models that allows for an efficient distributed algorithm. This is done by exploiting the decoupling of the data given the inducing points to re-formulate the evidence lower bound in a Map-Reduce setting. We show that the inference scales well with data and computational resources, while preserving a balanced distribution of the load among the nodes. We further demonstrate the utility in scaling Gaussian processes to big data. We show that GP performance improves with increasing amounts of data in regression (on flight data with 2 million records) and latent variable modelling (on MNIST). The results show that GPs perform better than many common models often used for big data.

9 pages, 8 figures

References in corpus (1)

• ClusterCluster: Parallel Markov Chain Monte Carlo for Dirichlet Process Mixtures

Cited by in corpus (20)

• Scalable Variational Gaussian Process Classification
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• Dropout as a Bayesian Approximation: Appendix
• MCMC for Variationally Sparse Gaussian Processes
• Gaussian Process Prior Variational Autoencoders
• Latent Gaussian Processes for Distribution Estimation of Multivariate Categorical Data
• FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing
• Deep Structured Mixtures of Gaussian Processes
• Gaussian Process Random Fields
• Large-scale Heteroscedastic Regression via Gaussian Process
• Stochastic Variational Inference for Bayesian Sparse Gaussian Process Regression
• Ensemble Kalman Filtering for Online Gaussian Process Regression and Learning
• Modular Gaussian Processes for Transfer Learning
• Recent Advances in Data-Driven Wireless Communication Using Gaussian Processes: A Comprehensive Survey
• Deep Recurrent Gaussian Process with Variational Sparse Spectrum Approximation
• Deep Latent-Variable Kernel Learning
• Identifying Reliable Annotations for Large Scale Image Segmentation
• Healing Products of Gaussian Processes
• Recyclable Gaussian Processes
• Online Anomaly Detection with Sparse Gaussian Processes