On the Similarity between the Laplace and Neural Tangent Kernels
arXiv:2007.01580
Abstract
Recent theoretical work has shown that massively overparameterized neural networks are equivalent to kernel regressors that use Neural Tangent Kernels(NTK). Experiments show that these kernel methods perform similarly to real neural networks. Here we show that NTK for fully connected networks is closely related to the standard Laplace kernel. We show theoretically that for normalized data on the hypersphere both kernels have the same eigenfunctions and their eigenvalues decay polynomially at the same rate, implying that their Reproducing Kernel Hilbert Spaces (RKHS) include the same sets of functions. This means that both kernels give rise to classes of functions with the same smoothness properties. The two kernels differ for data off the hypersphere, but experiments indicate that when data is properly normalized these differences are not significant. Finally, we provide experiments on real data comparing NTK and the Laplace kernel, along with a larger class ofγ-exponential kernels. We show that these perform almost identically. Our results suggest that much insight about neural networks can be obtained from analysis of the well-known Laplace kernel, which has a simple closed-form.
References in corpus (16)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Searching for Exotic Particles in High-Energy Physics with Deep Learning
- A Convergence Theory for Deep Learning via Over-Parameterization
- Deep Neural Networks as Gaussian Processes
- On Exact Computation with an Infinitely Wide Neural Net
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences
- Convolutional Kernel Networks
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
- The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
- To understand deep learning we need to understand kernel learning
- Enhanced Convolutional Neural Tangent Kernels
- Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
- Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks