Linear Multiple Low-Rank Kernel Based Stationary Gaussian Processes Regression for Time Series

Abstract

Gaussian processes (GP) for machine learning have been studied systematically over the past two decades and they are by now widely used in a number of diverse applications. However, GP kernel design and the associated hyper-parameter optimization are still hard and to a large extend open problems. In this paper, we consider the task of GP regression for time series modeling and analysis. The underlying stationary kernel can be approximated arbitrarily close by a new proposed grid spectral mixture (GSM) kernel, which turns out to be a linear combination of low-rank sub-kernels. In the case where a large number of the sub-kernels are used, either the Nyström or the random Fourier feature approximations can be adopted to deal efficiently with the computational demands. The unknown GP hyper-parameters consist of the non-negative weights of all sub-kernels as well as the noise variance; their estimation is performed via the maximum-likelihood (ML) estimation framework. Two efficient numerical optimization methods for solving the unknown hyper-parameters are derived, including a sequential majorization-minimization (MM) method and a non-linearly constrained alternating direction of multiplier method (ADMM). The MM matches perfectly with the proven low-rank property of the proposed GSM sub-kernels and turns out to be a part of efficiency, stable, and efficient solver, while the ADMM has the potential to generate better local minimum in terms of the test MSE. Experimental results, based on various classic time series data sets, corroborate that the proposed GSM kernel-based GP regression model outperforms several salient competitors of similar kind in terms of prediction mean-squared-error and numerical stability.

arXiv comment15 pages, 5 figures, submitted

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Practical Bayesian Optimization of Machine Learning Algorithms
• Deep Neural Networks as Gaussian Processes
• Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification
• Structure Discovery in Nonparametric Regression through Compositional Kernel Search
• Gaussian Process Kernels for Pattern Discovery and Extrapolation
• Wireless Traffic Prediction with Scalable Gaussian Process: Framework, Algorithms, and Verification
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Online Distributed Learning Over Networks in RKH Spaces Using Random Fourier Features
• Bayesian Nonparametric Kernel-Learning
• AutoGP: Exploring the Capabilities and Limitations of Gaussian Process Models
• Marginalizing Gaussian Process Hyperparameters using Sequential Monte Carlo
• MCMC methods for Gaussian process models using fast approximations for the likelihood

Cited by in corpus (2)

• Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling
• Recent Advances in Data-Driven Wireless Communication Using Gaussian Processes: A Comprehensive Survey