Stochastic Variational Deep Kernel Learning

Abstract

Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task learning, additive covariance structures, and stochastic gradient training. Specifically, we apply additive base kernels to subsets of output features from deep neural architectures, and jointly learn the parameters of the base kernels and deep network through a Gaussian process marginal likelihood objective. Within this framework, we derive an efficient form of stochastic variational inference which leverages local kernel interpolation, inducing points, and structure exploiting algebra. We show improved performance over stand alone deep networks, SVMs, and state of the art scalable Gaussian processes on several classification benchmarks, including an airline delay dataset containing 6 million training points, CIFAR, and ImageNet.

13 pages, 6 tables, 3 figures. Appearing in NIPS 2016

Cited by in corpus (59)

• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian optimization of a free-electron laser
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty
• Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• A Fine-Grained Spectral Perspective on Neural Networks
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• The Functional Neural Process
• Exact Gaussian Processes on a Million Data Points
• Uncertainty Quantification and Deep Ensembles
• Kernel methods through the roof: handling billions of points efficiently
• Recurrent Attentive Neural Process for Sequential Data
• Multimodal Word Distributions
• Deep State-Space Gaussian Processes
• Causal Inference for Time series Analysis: Problems, Methods and Evaluation
• Constant-Time Predictive Distributions for Gaussian Processes
• Deep Gaussian Processes: A Survey
• GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning
• FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing
• Learnable Embedding Space for Efficient Neural Architecture Compression
• A Gaussian Process perspective on Convolutional Neural Networks
• Improving Robustness and Uncertainty Modelling in Neural Ordinary Differential Equations
• Personalized Federated Learning with Gaussian Processes
• Being Bayesian about Categorical Probability
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes
• Infinitely Wide Graph Convolutional Networks: Semi-supervised Learning via Gaussian Processes
• Variational Implicit Processes
• No-Trick (Treat) Kernel Adaptive Filtering using Deterministic Features
• Calibrated simplex-mapping classification
• McKernel: A Library for Approximate Kernel Expansions in Log-linear Time
• Online tuning and light source control using a physics-informed Gaussian process Adi
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Deep Sigma Point Processes
• Energy-Based Processes for Exchangeable Data
• An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence
• Non-Parametric Calibration for Classification
• Few-shot Learning for Spatial Regression
• Recent Advances in Data-Driven Wireless Communication Using Gaussian Processes: A Comprehensive Survey
• Last Layer Marginal Likelihood for Invariance Learning
• Uncertainty-aware Remaining Useful Life predictor
• Kernelized Capsule Networks
• Efficient Transfer Bayesian Optimization with Auxiliary Information
• Multiview Variational Deep Learning with Application to Scalable Indoor Localization
• Longitudinal Deep Kernel Gaussian Process Regression
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data
• Deep Latent-Variable Kernel Learning
• End-to-End Learning of Deep Kernel Acquisition Functions for Bayesian Optimization
• Deep Kernel Learning for Mortality Prediction in the Face of Temporal Shift
• Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits
• Deep kernel learning for integral measurements
• Deep Kernel Learning for Clustering
• Efficient Correlated Topic Modeling with Topic Embedding
• Hierarchical Inducing Point Gaussian Process for Inter-domain Observations
• Generative Particle Variational Inference via Estimation of Functional Gradients
• Hierarchical Non-Stationary Temporal Gaussian Processes With $L^1$-Regularization
• GP-ConvCNP: Better Generalization for Convolutional Conditional Neural Processes on Time Series Data
• Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations