Non-asymptotic approximations of neural networks by Gaussian processes
arXiv:2102.08668
Abstract
We study the extent to which wide neural networks may be approximated by Gaussian processes when initialized with random weights. It is a well-established fact that as the width of a network goes to infinity, its law converges to that of a Gaussian process. We make this quantitative by establishing explicit convergence rates for the central limit theorem in an infinite-dimensional functional space, metrized with a natural transportation distance. We identify two regimes of interest; when the activation function is polynomial, its degree determines the rate of convergence, while for non-polynomial activations, the rate is governed by the smoothness of the function.
arXiv comment18 pages
References in corpus (15)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- On Lazy Training in Differentiable Programming
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Scaling description of generalization with number of parameters in deep learning
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Steps Toward Deep Kernel Methods from Infinite Neural Networks
- Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
- Stein's method for normal approximation in Wasserstein distances with application to the multivariate Central Limit Theorem
- Finite size corrections for neural network Gaussian processes
- Asymptotics of Wide Convolutional Neural Networks
- Non-Gaussian processes and neural networks at finite widths
- On the asymptotics of wide networks with polynomial activations
- A CLT in Stein's distance for generalized Wishart matrices and higher order tensors