Sequential Gaussian Processes for Online Learning of Nonstationary Functions

Abstract

Many machine learning problems can be framed in the context of estimating functions, and often these are time-dependent functions that are estimated in real-time as observations arrive. Gaussian processes (GPs) are an attractive choice for modeling real-valued nonlinear functions due to their flexibility and uncertainty quantification. However, the typical GP regression model suffers from several drawbacks: 1) Conventional GP inference scales $O(N^3)$ with respect to the number of observations; 2) Updating a GP model sequentially is not trivial; and 3) Covariance kernels typically enforce stationarity constraints on the function, while GPs with non-stationary covariance kernels are often intractable to use in practice. To overcome these issues, we propose a sequential Monte Carlo algorithm to fit infinite mixtures of GPs that capture non-stationary behavior while allowing for online, distributed inference. Our approach empirically improves performance over state-of-the-art methods for online GP estimation in the presence of non-stationarity in time-series data. To demonstrate the utility of our proposed online Gaussian process mixture-of-experts approach in applied settings, we show that we can sucessfully implement an optimization algorithm using online Gaussian process bandits.

References in corpus (7)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Bayesian Online Changepoint Detection
• Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
• Slice sampling covariance hyperparameters of latent Gaussian models
• An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection
• Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier
• Deep Structured Mixtures of Gaussian Processes

Cited by in corpus (1)

• Continual Multi-task Gaussian Processes