Dynamically Stable Infinite-Width Limits of Neural Classifiers
arXiv:2006.06574
Abstract
Recent research has been focused on two different approaches to studying neural networks training in the limit of infinite width (1) a mean-field (MF) and (2) a constant neural tangent kernel (NTK) approximations. These two approaches have different scaling of hyperparameters with the width of a network layer and as a result, different infinite-width limit models. We propose a general framework to study how the limit behavior of neural models depends on the scaling of hyperparameters with network width. Our framework allows us to derive scaling for existing MF and NTK limits, as well as an uncountable number of other scalings that lead to a dynamically stable limit behavior of corresponding models. However, only a finite number of distinct limit models are induced by these scalings. Each distinct limit model corresponds to a unique combination of such properties as boundedness of logits and tangent kernels at initialization or stationarity of tangent kernels. Existing MF and NTK limit models, as well as one novel limit model, satisfy most of the properties demonstrated by finite-width models. We also propose a novel initialization-corrected mean-field limit that satisfies all properties noted above, and its corresponding model is a simple modification for a finite-width model.
arXiv comment26 pages, 7 figures
References in corpus (10)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- On Exact Computation with an Infinitely Wide Neural Net
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Kernel and Rich Regimes in Overparametrized Models
- Asymptotics of Wide Networks from Feynman Diagrams
- Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
- Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
- Collective evolution of weights in wide neural networks
- Towards a General Theory of Infinite-Width Limits of Neural Classifiers