Deep Latent-Variable Kernel Learning
arXiv:2005.08467
Abstract
Deep kernel learning (DKL) leverages the connection between Gaussian process (GP) and neural networks (NN) to build an end-to-end, hybrid model. It combines the capability of NN to learn rich representations under massive data and the non-parametric property of GP to achieve automatic regularization that incorporates a trade-off between model fit and model complexity. However, the deterministic encoder may weaken the model regularization of the following GP part, especially on small datasets, due to the free latent representation. We therefore present a complete deep latent-variable kernel learning (DLVKL) model wherein the latent variables perform stochastic encoding for regularized representation. We further enhance the DLVKL from two aspects: (i) the expressive variational posterior through neural stochastic differential equation (NSDE) to improve the approximation quality, and (ii) the hybrid prior taking knowledge from both the SDE prior and the posterior to arrive at a flexible trade-off. Intensive experiments imply that the DLVKL-NSDE performs similarly to the well calibrated GP on small datasets, and outperforms existing deep GPs on large datasets.
arXiv comment13 pages, 8 figures, preprint under review
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Conditional Image Generation with PixelCNN Decoders
- Variational Inference with Normalizing Flows
- Deep Neural Networks as Gaussian Processes
- Gaussian Processes for Big Data
- Deep Variational Information Bottleneck
- Variational Lossy Autoencoder
- Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)
- Stochastic Variational Deep Kernel Learning
- Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
- Avoiding pathologies in very deep networks
- Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models
- Deep learning with differential Gaussian process flows
- Scalable Gaussian Process Classification with Additive Noise for Various Likelihoods