Large-scale Heteroscedastic Regression via Gaussian Process
arXiv:1811.01179
Abstract
Heteroscedastic regression considering the varying noises among observations has many applications in the fields like machine learning and statistics. Here we focus on the heteroscedastic Gaussian process (HGP) regression which integrates the latent function and the noise function together in a unified non-parametric Bayesian framework. Though showing remarkable performance, HGP suffers from the cubic time complexity, which strictly limits its application to big data. To improve the scalability, we first develop a variational sparse inference algorithm, named VSHGP, to handle large-scale datasets. Furthermore, two variants are developed to improve the scalability and capability of VSHGP. The first is stochastic VSHGP (SVSHGP) which derives a factorized evidence lower bound, thus enhancing efficient stochastic variational inference. The second is distributed VSHGP (DVSHGP) which (i) follows the Bayesian committee machine formalism to distribute computations over multiple local VSHGP experts with many inducing points; and (ii) adopts hybrid parameters for experts to guard against over-fitting and capture local variety. The superiority of DVSHGP and SVSHGP as compared to existing scalable heteroscedastic/homoscedastic GPs is then extensively verified on various datasets.
arXiv comment14 pages, 15 figures
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- Gaussian Processes for Big Data
- Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
- Practical heteroskedastic Gaussian process modeling for large simulation experiments
- GPz: Non-stationary sparse Gaussian processes for heteroscedastic uncertainty estimation in photometric redshifts
- Variational Fourier features for Gaussian processes
- Information Directed Sampling and Bandits with Heteroscedastic Noise
- Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models
- Non-Stationary Gaussian Process Regression with Hamiltonian Monte Carlo
- Nested Variational Compression in Deep Gaussian Processes
- Non-Stationary Spectral Kernels
- Gaussian Process Conditional Density Estimation
- Modelling local and global phenomena with sparse Gaussian processes
- Parametric Gaussian Process Regressors
- Stochastic Variational Inference for Bayesian Sparse Gaussian Process Regression