On the asymptotics of wide networks with polynomial activations

Abstract

We consider an existing conjecture addressing the asymptotic behavior of neural networks in the large width limit. The results that follow from this conjecture include tight bounds on the behavior of wide networks during stochastic gradient descent, and a derivation of their finite-width dynamics. We prove the conjecture for deep networks with polynomial activation functions, greatly extending the validity of these results. Finally, we point out a difference in the asymptotic behavior of networks with analytic (and non-linear) activation functions and those with piecewise-linear activations such as ReLU.

arXiv comment8+12 pages, 6 figures, 2 tables

References in corpus (7)

• Wide Residual Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• Asymptotics of Wide Networks from Feynman Diagrams
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Non-Gaussian processes and neural networks at finite widths
• On Infinite-Width Hypernetworks

Cited by in corpus (7)

• On the training dynamics of deep networks with $L_2$ regularization
• Asymptotics of representation learning in finite Bayesian neural networks
• Asymptotics of Wide Convolutional Neural Networks
• Exact marginal prior distributions of finite Bayesian neural networks
• Non-asymptotic approximations of neural networks by Gaussian processes
• Rate of Convergence of Polynomial Networks to Gaussian Processes
• Neural Networks as Kernel Learners: The Silent Alignment Effect