Bias-Free Scalable Gaussian Processes via Randomized Truncations
arXiv:2102.06695
Abstract
Scalable Gaussian Process methods are computationally attractive, yet introduce modeling biases that require rigorous study. This paper analyzes two common techniques: early truncated conjugate gradients (CG) and random Fourier features (RFF). We find that both methods introduce a systematic bias on the learned hyperparameters: CG tends to underfit while RFF tends to overfit. We address these issues using randomized truncation estimators that eliminate bias in exchange for increased variance. In the case of RFF, we show that the bias-to-variance conversion is indeed a trade-off: the additional variance proves detrimental to optimization. However, in the case of CG, our unbiased learning procedure meaningfully outperforms its biased counterpart with minimal additional computation.
References in corpus (9)
- GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
- Gaussian Processes for Big Data
- Residual Flows for Invertible Generative Modeling
- Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions
- On Russian Roulette Estimates for Bayesian Inference with Doubly-Intractable Likelihoods
- Preconditioning Kernel Matrices
- Enabling scalable stochastic gradient-based inference for Gaussian processes by employing the Unbiased LInear System SolvEr (ULISSE)
- Efficient Optimization of Loops and Limits with Randomized Telescoping Sums
- Randomized Automatic Differentiation