Conditional Deep Gaussian Processes: multi-fidelity kernel learning

Abstract

Deep Gaussian Processes (DGPs) were proposed as an expressive Bayesian model capable of a mathematically grounded estimation of uncertainty. The expressivity of DPGs results from not only the compositional character but the distribution propagation within the hierarchy. Recently, [1] pointed out that the hierarchical structure of DGP well suited modeling the multi-fidelity regression, in which one is provided sparse observations with high precision and plenty of low fidelity observations. We propose the conditional DGP model in which the latent GPs are directly supported by the fixed lower fidelity data. Then the moment matching method in [2] is applied to approximate the marginal prior of conditional DGP with a GP. The obtained effective kernels are implicit functions of the lower-fidelity data, manifesting the expressivity contributed by distribution propagation within the hierarchy. The hyperparameters are learned via optimizing the approximate marginal likelihood. Experiments with synthetic and high dimensional data show comparable performance against other multi-fidelity regression methods, variational inference, and multi-output GP. We conclude that, with the low fidelity data and the hierarchical DGP structure, the effective kernel encodes the inductive bias for true function allowing the compositional freedom discussed in [3,4].

Submitted on Oct 1st 2021 with a sequel titled "Conditional Deep Gaussian Process: empirical Bayes hyperdata learning"

References in corpus (17)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Convolutional Kernel Networks
• Functional Regularisation for Continual Learning with Gaussian Processes
• Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
• Doubly Stochastic Variational Inference for Deep Gaussian Processes
• Differentiable Compositional Kernel Learning for Gaussian Processes
• Deep Gaussian Processes for Multi-fidelity Modeling
• Spectral Mixture Kernels for Multi-Output Gaussian Processes
• Deep Gaussian Processes with Importance-Weighted Variational Inference
• Scalable Exact Inference in Multi-Output Gaussian Processes
• The Gaussian Process Autoregressive Regression Model (GPAR)
• Implicit Posterior Variational Inference for Deep Gaussian Processes
• Interpretable deep Gaussian processes with moments
• Learning spectrograms with convolutional spectral kernels
• Compositional uncertainty in deep Gaussian processes
• Multi-Fidelity High-Order Gaussian Processes for Physical Simulation
• Bayesian Alignments of Warped Multi-Output Gaussian Processes