On neural network kernels and the storage capacity problem
arXiv:2201.04669 · doi:10.1162/neco_a_01494
Abstract
In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width Neural Network Gaussian Process Kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.
arXiv comment5 pages, no figures
References in corpus (8)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Deep Neural Networks as Gaussian Processes
- Exponential expressivity in deep neural networks through transient chaos
- Gaussian Process Behaviour in Wide Deep Neural Networks
- Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
- Asymptotics of representation learning in finite Bayesian neural networks
- Activation function dependence of the storage capacity of treelike neural networks
- Effect of Activation Functions on the Training of Overparametrized Neural Nets