On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures
arXiv:2006.13645
Abstract
The Neural Tangent Kernel (NTK) is an important milestone in the ongoing effort to build a theory for deep learning. Its prediction that sufficiently wide neural networks behave as kernel methods, or equivalently as random feature models, has been confirmed empirically for certain wide architectures. It remains an open question how well NTK theory models standard neural network architectures of widths common in practice, trained on complex datasets such as ImageNet. We study this question empirically for two well-known convolutional neural network architectures, namely AlexNet and LeNet, and find that their behavior deviates significantly from their finite-width NTK counterparts. For wider versions of these networks, where the number of channels and widths of fully-connected layers are increased, the deviation decreases.
arXiv comment10 pages
References in corpus (9)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Wide Residual Networks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- One weird trick for parallelizing convolutional neural networks
- On Lazy Training in Differentiable Programming
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
- Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
- Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks