Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning

Abstract

It is desirable to combine the expressive power of deep learning with Gaussian Process (GP) in one expressive Bayesian learning model. Deep kernel learning showed success in adopting a deep network for feature extraction followed by a GP used as function model. Recently,it was suggested that, albeit training with marginal likelihood, the deterministic nature of feature extractor might lead to overfitting while the replacement with a Bayesian network seemed to cure it. Here, we propose the conditional Deep Gaussian Process (DGP) in which the intermediate GPs in hierarchical composition are supported by the hyperdata and the exposed GP remains zero mean. Motivated by the inducing points in sparse GP, the hyperdata also play the role of function supports, but are hyperparameters rather than random variables. We follow our previous moment matching approach to approximate the marginal prior for conditional DGP with a GP carrying an effective kernel. Thus, as in empirical Bayes, the hyperdata are learned by optimizing the approximate marginal likelihood which implicitly depends on the hyperdata via the kernel. We shall show the equivalence with the deep kernel learning in the limit of dense hyperdata in latent space. However, the conditional DGP and the corresponding approximate inference enjoy the benefit of being more Bayesian than deep kernel learning. Preliminary extrapolation results demonstrate expressive power from the depth of hierarchy by exploiting the exact covariance and hyperdata learning, in comparison with GP kernel composition, DGP variational inference and deep kernel learning. We also address the non-Gaussian aspect of our model as well as way of upgrading to a full Bayes inference.

arXiv commentAccepted by Special Issue "Probabilistic Methods for Deep Learning" of Entropy, 15 pages, reference to recent papers of finite Bayesian neural network added

References in corpus (24)

• Auto-Encoding Variational Bayes
• Expectation Propagation for approximate Bayesian inference
• Generative Moment Matching Networks
• Structure Discovery in Nonparametric Regression through Compositional Kernel Search
• Learning Functions: When Is Deep Better Than Shallow
• Learning from Distributions via Support Measure Machines
• Random Feature Expansions for Deep Gaussian Processes
• Avoiding pathologies in very deep networks
• Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
• Doubly Stochastic Variational Inference for Deep Gaussian Processes
• Differentiable Compositional Kernel Learning for Gaussian Processes
• Understanding Priors in Bayesian Neural Networks at the Unit Level
• Deep Gaussian Processes with Importance-Weighted Variational Inference
• Sparse Gaussian Processes with Spherical Harmonic Features
• Implicit Posterior Variational Inference for Deep Gaussian Processes
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Interpretable deep Gaussian processes with moments
• Non-Gaussian processes and neural networks at finite widths
• Exact marginal prior distributions of finite Bayesian neural networks
• Sparse Orthogonal Variational Inference for Gaussian Processes
• Compositional uncertainty in deep Gaussian processes
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Characterizing Deep Gaussian Processes via Nonlinear Recurrence Systems