Modulating Scalable Gaussian Processes for Expressive Statistical Learning

Abstract

For a learning task, Gaussian process (GP) is interested in learning the statistical relationship between inputs and outputs, since it offers not only the prediction mean but also the associated variability. The vanilla GP however struggles to learn complicated distribution with the property of, e.g., heteroscedastic noise, multi-modality and non-stationarity, from massive data due to the Gaussian marginal and the cubic complexity. To this end, this article studies new scalable GP paradigms including the non-stationary heteroscedastic GP, the mixture of GPs and the latent GP, which introduce additional latent variables to modulate the outputs or inputs in order to learn richer, non-Gaussian statistical representation. We further resort to different variational inference strategies to arrive at analytical or tighter evidence lower bounds (ELBOs) of the marginal likelihood for efficient and effective model training. Extensive numerical experiments against state-of-the-art GP and neural network (NN) counterparts on various tasks verify the superiority of these scalable modulated GPs, especially the scalable latent GP, for learning diverse data distributions.

arXiv comment31 pages, 9 figures, 4 tables, preprint under review

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• Categorical Reparameterization with Gumbel-Softmax
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Gaussian Processes for Big Data
• Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
• GPz: Non-stationary sparse Gaussian processes for heteroscedastic uncertainty estimation in photometric redshifts
• Importance Weighting and Variational Inference
• Conditional Variational Autoencoder for Neural Machine Translation
• Gaussian Process Regression with Heteroscedastic or Non-Gaussian Residuals
• Exact Gaussian Processes on a Million Data Points
• Variational Inference for Uncertainty on the Inputs of Gaussian Process Models
• Deep Gaussian Processes with Importance-Weighted Variational Inference
• Latent Gaussian Process Regression
• Decomposing feature-level variation with Covariate Gaussian Process Latent Variable Models
• Deep Latent-Variable Kernel Learning