Are wider nets better given the same number of parameters?
arXiv:2010.14495
Abstract
Empirical studies demonstrate that the performance of neural networks improves with increasing number of parameters. In most of these studies, the number of parameters is increased by increasing the network width. This begs the question: Is the observed improvement due to the larger number of parameters, or is it due to the larger width itself? We compare different ways of increasing model width while keeping the number of parameters constant. We show that for models initialized with a random, static sparsity pattern in the weight tensors, network width is the determining factor for good performance, while the number of weights is secondary, as long as trainability is ensured. As a step towards understanding this effect, we analyze these models in the framework of Gaussian Process kernels. We find that the distance between the sparse finite-width model kernel and the infinite-width kernel at initialization is indicative of model performance.
9 pages
References in corpus (14)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Scaling Laws for Neural Language Models
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Deep Neural Networks as Gaussian Processes
- The State of Sparsity in Deep Neural Networks
- Deep Learning Scaling is Predictable, Empirically
- On Exact Computation with an Infinitely Wide Neural Net
- Deep Rewiring: Training very sparse deep networks
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Kernel and Rich Regimes in Overparametrized Models
- Pruning Neural Networks at Initialization: Why are We Missing the Mark?
- Finite Versus Infinite Neural Networks: an Empirical Study