Finite Depth and Width Corrections to the Neural Tangent Kernel

Abstract

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not deterministic if depth and width simultaneously tend to infinity. Moreover, we prove that for such deep and wide networks, the NTK has a non-trivial evolution during training by showing that the mean of its first SGD update is also exponential in the ratio of network depth to width. This is sharp contrast to the regime where depth is fixed and network width is very large. Our results suggest that, unlike relatively shallow and wide networks, deep and wide ReLU networks are capable of learning data-dependent features even in the so-called lazy training regime.

arXiv comment27 pages, 2 figures, comments welcome

References in corpus (8)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• On the Inductive Bias of Neural Tangent Kernels
• Asymptotics of Wide Networks from Feynman Diagrams

Cited by in corpus (9)

• Scaling description of generalization with number of parameters in deep learning
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• On the asymptotics of wide networks with polynomial activations
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• Predicting the outputs of finite deep neural networks trained with noisy gradients
• Neural Optimization Kernel: Towards Robust Deep Learning