Direct loss minimization algorithms for sparse Gaussian processes

Abstract

The paper provides a thorough investigation of Direct loss minimization (DLM), which optimizes the posterior to minimize predictive loss, in sparse Gaussian processes. For the conjugate case, we consider DLM for log-loss and DLM for square loss showing a significant performance improvement in both cases. The application of DLM in non-conjugate cases is more complex because the logarithm of expectation in the log-loss DLM objective is often intractable and simple sampling leads to biased estimates of gradients. The paper makes two technical contributions to address this. First, a new method using product sampling is proposed, which gives unbiased estimates of gradients (uPS) for the objective function. Second, a theoretical analysis of biased Monte Carlo estimates (bMC) shows that stochastic gradient descent converges despite the biased gradients. Experiments demonstrate empirical success of DLM. A comparison of the sampling methods shows that, while uPS is potentially more sample-efficient, bMC provides a better tradeoff in terms of convergence time and computational efficiency.

arXiv comment31 pages, 16 figures

References in corpus (10)

• Gaussian Processes for Big Data
• Scalable Variational Gaussian Process Classification
• Frequentist Consistency of Variational Bayes
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Variational Inference based on Robust Divergences
• Orthogonally Decoupled Variational Gaussian Processes
• Robust Deep Gaussian Processes
• Computing the quality of the Laplace approximation
• Monte Carlo Structured SVI for Two-Level Non-Conjugate Models