Gaussian Process Behaviour in Wide Deep Neural Networks

Abstract

Whilst deep neural networks have shown great empirical success, there is still much work to be done to understand their theoretical properties. In this paper, we study the relationship between random, wide, fully connected, feedforward networks with more than one hidden layer and Gaussian processes with a recursive kernel definition. We show that, under broad conditions, as we make the architecture increasingly wide, the implied random function converges in distribution to a Gaussian process, formalising and extending existing results by Neal (1996) to deep networks. To evaluate convergence rates empirically, we use maximum mean discrepancy. We then compare finite Bayesian deep networks from the literature to Gaussian processes in terms of the key predictive quantities of interest, finding that in some cases the agreement can be very close. We discuss the desirability of Gaussian process behaviour and review non-Gaussian alternative models from the literature.

arXiv commentThis work substantially extends the work of Matthews et al. (2018) published at the International Conference on Learning Representations (ICLR) 2018

Cited by in corpus (17)

• The Modern Mathematics of Deep Learning
• Neural Networks and Quantum Field Theory
• Linear Multiple Low-Rank Kernel Based Stationary Gaussian Processes Regression for Time Series
• Quantum field-theoretic machine learning
• Nonperturbative renormalization for the neural network-QFT correspondence
• Deeper Connections between Neural Networks and Gaussian Processes Speed-up Active Learning
• Depth induces scale-averaging in overparameterized linear Bayesian neural networks
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes
• On neural network kernels and the storage capacity problem
• Infinite Neural Network Quantum States: Entanglement and Training Dynamics
• Forecasting temporal variation of aftershocks immediately after a main shock using Gaussian process regression
• Accelerating the training of single-layer binary neural networks using the HHL quantum algorithm
• Bayesian RG Flow in Neural Network Field Theories
• Uncertainty Quantification From Scaling Laws in Deep Neural Networks
• The Renyi Gaussian Process: Towards Improved Generalization
• Posterior contraction rates for constrained deep Gaussian processes in density estimation and classication
• Discriminative Clustering with Representation Learning with any Ratio of Labeled to Unlabeled Data