Deep Neural Networks as Point Estimates for Deep Gaussian Processes

Abstract

Neural networks and Gaussian processes are complementary in their strengths and weaknesses. Having a better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on interpreting activation functions as interdomain inducing features through a rigorous analysis of the interplay between activation functions and kernels. This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datasets.

arXiv comment35th Conference on Neural Information Processing Systems (NeurIPS 2021)

References in corpus (18)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Gaussian Processes for Big Data
• Scalable Variational Gaussian Process Classification
• Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Random Feature Expansions for Deep Gaussian Processes
• Variational Fourier features for Gaussian processes
• Avoiding pathologies in very deep networks
• A Framework for Interdomain and Multioutput Gaussian Processes
• Nested Variational Compression in Deep Gaussian Processes
• Learning Invariances using the Marginal Likelihood
• Sparse Gaussian Processes with Spherical Harmonic Features
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• GPflux: A Library for Deep Gaussian Processes
• Inter-domain Deep Gaussian Processes