Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo

Abstract

Deep Gaussian Processes (DGPs) are hierarchical generalizations of Gaussian Processes that combine well calibrated uncertainty estimates with the high flexibility of multilayer models. One of the biggest challenges with these models is that exact inference is intractable. The current state-of-the-art inference method, Variational Inference (VI), employs a Gaussian approximation to the posterior distribution. This can be a potentially poor unimodal approximation of the generally multimodal posterior. In this work, we provide evidence for the non-Gaussian nature of the posterior and we apply the Stochastic Gradient Hamiltonian Monte Carlo method to generate samples. To efficiently optimize the hyperparameters, we introduce the Moving Window MCEM algorithm. This results in significantly better predictions at a lower computational cost than its VI counterpart. Thus our method establishes a new state-of-the-art for inference in DGPs.

Cited by in corpus (24)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• Causal Inference for Time series Analysis: Problems, Methods and Evaluation
• Bayesian Optimization using Deep Gaussian Processes
• Deep Gaussian Processes: A Survey
• Probabilistic Metamodels for an Efficient Characterization of Complex Driving Scenarios
• Icebreaker: Element-wise Active Information Acquisition with Bayesian Deep Latent Gaussian Model
• Parametric Gaussian Process Regressors
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Compositional uncertainty in deep Gaussian processes
• Sample-efficient reinforcement learning using deep Gaussian processes
• Bayesian Image Classification with Deep Convolutional Gaussian Processes
• Monotonic Gaussian Process Flow
• Stein Variational Gaussian Processes
• Deep Sigma Point Processes
• Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties
• On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes
• Inter-domain Deep Gaussian Processes
• Conditional Deep Gaussian Processes: multi-fidelity kernel learning
• Learning GPLVM with arbitrary kernels using the unscented transformation
• Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Features
• Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning
• Convolutional Normalizing Flows for Deep Gaussian Processes