Subaging in underparametrized Deep Neural Networks
arXiv:2209.02517 · doi:10.1088/2632-2153/ac8f1b
Abstract
We consider a simple classification problem to show that the dynamics of finite-width Deep Neural Networks in the underparametrized regime gives rise to effects similar to those associated with glassy systems, namely a slow evolution of the loss function and aging. Remarkably, the aging is sublinear in the waiting time (subaging) and the power-law exponent characterizing it is robust to different architectures under the constraint of a constant total number of parameters. Our results are maintained in the more complex scenario of the MNIST database. We find that for this database there is a unique exponent ruling the subaging behavior in the whole phase.
arXiv comment16 pages, 10 figures. Manuscript accepted in"Machine Learning: Science and Technology"
References in corpus (20)
- Machine learning and the physical sciences
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A high-bias, low-variance introduction to Machine Learning for physicists
- The Loss Surfaces of Multilayer Networks
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Ageing and Rheology in Soft Materials
- Three Factors Influencing Minima in SGD
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- The simplest model of jamming
- The jamming transition as a paradigm to understand the loss landscape of deep neural networks
- Multiple scaling regimes in simple aging models
- A jamming transition from under- to over-parametrization affects loss landscape and generalization
- Rejuvenation and Memory Effects in a Structural Glass
- Comparing Dynamics: Deep Neural Networks versus Glassy Systems
- Critical scaling and aging in cooling systems near the jamming transition
- Glasses and aging: A Statistical Mechanics Perspective
- Sub-aging in a Domain Growth Model
- How noise affects the Hessian spectrum in overparameterized neural networks
- Aging in glassy systems: new experiments, simple models, and open questions
- Phases of learning dynamics in artificial neural networks: with or without mislabeled data