Compositional uncertainty in deep Gaussian processes

Abstract

Gaussian processes (GPs) are nonparametric priors over functions. Fitting a GP implies computing a posterior distribution of functions consistent with the observed data. Similarly, deep Gaussian processes (DGPs) should allow us to compute a posterior distribution of compositions of multiple functions giving rise to the observations. However, exact Bayesian inference is intractable for DGPs, motivating the use of various approximations. We show that the application of simplifying mean-field assumptions across the hierarchy leads to the layers of a DGP collapsing to near-deterministic transformations. We argue that such an inference scheme is suboptimal, not taking advantage of the potential of the model to discover the compositional structure in the data. To address this issue, we examine alternative variational inference schemes allowing for dependencies across different layers and discuss their advantages and limitations.

17 pages

References in corpus (10)

• Auto-Encoding Variational Bayes
• Gaussian Processes for Big Data
• Functional Variational Bayesian Neural Networks
• Input Warping for Bayesian Optimization of Non-stationary Functions
• Variational Fourier features for Gaussian processes
• Avoiding pathologies in very deep networks
• Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
• Nested Variational Compression in Deep Gaussian Processes
• On Nesting Monte Carlo Estimators
• Inverse Reinforcement Learning via Deep Gaussian Process

Cited by in corpus (2)

• Conditional Deep Gaussian Processes: multi-fidelity kernel learning
• Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning