Stochastic Variational Deep Kernel Learning

Abstract

Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task learning, additive covariance structures, and stochastic gradient training. Specifically, we apply additive base kernels to subsets of output features from deep neural architectures, and jointly learn the parameters of the base kernels and deep network through a Gaussian process marginal likelihood objective. Within this framework, we derive an efficient form of stochastic variational inference which leverages local kernel interpolation, inducing points, and structure exploiting algebra. We show improved performance over stand alone deep networks, SVMs, and state of the art scalable Gaussian processes on several classification benchmarks, including an airline delay dataset containing 6 million training points, CIFAR, and ImageNet.

13 pages, 6 tables, 3 figures. Appearing in NIPS 2016

Cited by in corpus (18)

• Recurrent Attentive Neural Process for Sequential Data
• Causal Inference for Time series Analysis: Problems, Methods and Evaluation
• Deep Gaussian Processes: A Survey
• Learnable Embedding Space for Efficient Neural Architecture Compression
• Improving Robustness and Uncertainty Modelling in Neural Ordinary Differential Equations
• Personalized Federated Learning with Gaussian Processes
• Infinitely Wide Graph Convolutional Networks: Semi-supervised Learning via Gaussian Processes
• No-Trick (Treat) Kernel Adaptive Filtering using Deterministic Features
• Kernelized Capsule Networks
• Multiview Variational Deep Learning with Application to Scalable Indoor Localization
• Uncertainty-aware Remaining Useful Life predictor
• Deep Kernel Learning for Mortality Prediction in the Face of Temporal Shift
• Deep Latent-Variable Kernel Learning
• End-to-End Learning of Deep Kernel Acquisition Functions for Bayesian Optimization
• GP-ConvCNP: Better Generalization for Convolutional Conditional Neural Processes on Time Series Data
• Hierarchical Non-Stationary Temporal Gaussian Processes With $L^1$-Regularization
• Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations
• Efficient Correlated Topic Modeling with Topic Embedding