Approximate Inference Turns Deep Networks into Gaussian Processes

Abstract

Deep neural networks (DNN) and Gaussian processes (GP) are two powerful models with several theoretical connections relating them, but the relationship between their training methods is not well understood. In this paper, we show that certain Gaussian posterior approximations for Bayesian DNNs are equivalent to GP posteriors. This enables us to relate solutions and iterations of a deep-learning algorithm to GP inference. As a result, we can obtain a GP kernel and a nonlinear feature map while training a DNN. Surprisingly, the resulting kernel is the neural tangent kernel. We show kernels obtained on real datasets and demonstrate the use of the GP marginal likelihood to tune hyperparameters of DNNs. Our work aims to facilitate further research on combining DNNs and GPs in practical settings.

arXiv commentpublished at NeurIPS 2019: https://papers.nips.cc/paper/8573-approximate-inference-turns-deep-networks-into-gaussian-processes.pdf

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Practical Deep Learning with Bayesian Principles
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Provable Bayesian Inference via Particle Mirror Descent

Cited by in corpus (3)

• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees
• Fast Adaptation with Linearized Neural Networks