Deep kernel processes
arXiv:2010.01590
Abstract
We define deep kernel processes in which positive definite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions. Remarkably, we find that deep Gaussian processes (DGPs), Bayesian neural networks (BNNs), infinite BNNs, and infinite BNNs with bottlenecks can all be written as deep kernel processes. For DGPs the equivalence arises because the Gram matrix formed by the inner product of features is Wishart distributed, and as we show, standard isotropic kernels can be written entirely in terms of this Gram matrix -- we do not need knowledge of the underlying features. We define a tractable deep kernel process, the deep inverse Wishart process, and give a doubly-stochastic inducing-point variational inference scheme that operates on the Gram matrices, not on the features, as in DGPs. We show that the deep inverse Wishart process gives superior performance to DGPs and infinite BNNs on standard fully-connected baselines.
arXiv comment21 pages
References in corpus (12)
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Weight Uncertainty in Neural Networks
- Deep Neural Networks as Gaussian Processes
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Student-t Processes as Alternatives to Gaussian Processes
- Deep Convolutional Networks as shallow Gaussian Processes
- Generalised Wishart Processes
- PAC-Bayes with Backprop
- Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
- Stochastic Differential Equations with Variational Wishart Diffusions
- Why bigger is not always better: on finite and infinite neural networks