Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks

Abstract

Deep residual network architectures have been shown to achieve superior accuracy over classical feed-forward networks, yet their success is still not fully understood. Focusing on massively over-parameterized, fully connected residual networks with ReLU activation through their respective neural tangent kernels (ResNTK), we provide here a spectral analysis of these kernels. Specifically, we show that, much like NTK for fully connected networks (FC-NTK), for input distributed uniformly on the hypersphere ${\mathbb{S}}^{d-1}$, the eigenfunctions of ResNTK are the spherical harmonics and the eigenvalues decay polynomially with frequency $k$ as $k^{-d}$. These in turn imply that the set of functions in their Reproducing Kernel Hilbert Space are identical to those of FC-NTK, and consequently also to those of the Laplace kernel. We further show, by drawing on the analogy to the Laplace kernel, that depending on the choice of a hyper-parameter that balances between the skip and residual connections ResNTK can either become spiky with depth, as with FC-NTK, or maintain a stable shape.

References in corpus (10)

• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• On the Spectral Bias of Neural Networks
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
• Learning to Optimize Multigrid PDE Solvers
• Deep Equals Shallow for ReLU Networks in Kernel Regimes
• Interpolation and Learning with Scale Dependent Kernels
• Kernel-Based Smoothness Analysis of Residual Networks

Cited by in corpus (1)

• Neural Tangent Kernel Maximum Mean Discrepancy