Finite size corrections for neural network Gaussian processes

Abstract

There has been a recent surge of interest in modeling neural networks (NNs) as Gaussian processes. In the limit of a NN of infinite width the NN becomes equivalent to a Gaussian process. Here we demonstrate that for an ensemble of large, finite, fully connected networks with a single hidden layer the distribution of outputs at initialization is well described by a Gaussian perturbed by the fourth Hermite polynomial for weights drawn from a symmetric distribution. We show that the scale of the perturbation is inversely proportional to the number of units in the NN and that higher order terms decay more rapidly, thereby recovering the Edgeworth expansion. We conclude by observing that understanding how this perturbation changes under training would reveal the regimes in which the Gaussian process framework is valid to model NN behavior.

arXiv commentPresented at the 2019 ICML Workshop on Theoretical Physics for Deep Learning

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Weakly Non-Linear Gaussian Fluctuations and the Edgeworth Expansion
• Deep Convolutional Networks as shallow Gaussian Processes
• Renormalization Group and Probability Theory
• Generalized Central Limit Theorem and Renormalization Group
• On the use of the Edgeworth expansion in cosmology I: how to foresee and evade its pitfalls

Cited by in corpus (8)

• Neural Networks and Quantum Field Theory
• Finite Versus Infinite Neural Networks: an Empirical Study
• Asymptotics of representation learning in finite Bayesian neural networks
• Generalization bounds for deep learning
• Exact marginal prior distributions of finite Bayesian neural networks
• Non-asymptotic approximations of neural networks by Gaussian processes
• Bayesian RG Flow in Neural Network Field Theories
• A note on regularised NTK dynamics with an application to PAC-Bayesian training