On the infinite width limit of neural networks with a standard parameterization

Abstract

There are currently two parameterizations used to derive fixed kernels corresponding to infinite width neural networks, the NTK (Neural Tangent Kernel) parameterization and the naive standard parameterization. However, the extrapolation of both of these parameterizations to infinite width is problematic. The standard parameterization leads to a divergent neural tangent kernel while the NTK parameterization fails to capture crucial aspects of finite width networks such as: the dependence of training dynamics on relative layer widths, the relative training dynamics of weights and biases, and overall learning rate scale. Here we propose an improved extrapolation of the standard parameterization that preserves all of these properties as width is taken to infinity and yields a well-defined neural tangent kernel. We show experimentally that the resulting kernels typically achieve similar accuracy to those resulting from an NTK parameterization, but with better correspondence to the parameterization of typical finite width networks. Additionally, with careful tuning of width parameters, the improved standard parameterization kernels can outperform those stemming from an NTK parameterization. We release code implementing this improved standard parameterization as part of the Neural Tangents library at https://github.com/google/neural-tangents.

References in corpus (14)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• On the Inductive Bias of Neural Tangent Kernels
• Asymptotics of Wide Networks from Feynman Diagrams
• Random deep neural networks are biased towards simple functions
• The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
• Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Cited by in corpus (10)

• Finite Versus Infinite Neural Networks: an Empirical Study
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Feature Learning in Infinite-Width Neural Networks
• Towards NNGP-guided Neural Architecture Search
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• Dataset Meta-Learning from Kernel Ridge-Regression
• Exact posterior distributions of wide Bayesian neural networks
• Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology
• Comments on Leo Breiman's paper 'Statistical Modeling: The Two Cultures' (Statistical Science, 2001, 16(3), 199-231)