A jamming transition from under- to over-parametrization affects loss landscape and generalization

Abstract

We argue that in fully-connected networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. Under some general conditions, we show that this transition is sharp for the hinge loss. In the whole over-parametrized regime, poor minima of the loss are not encountered during training since the number of constraints to satisfy is too small to hamper minimization. Our findings support a link between this transition and the generalization properties of the network: as we increase the number of parameters of a given model, starting from an under-parametrized network, we observe that the generalization error displays three phases: (i) initial decay, (ii) increase until the transition point --- where it displays a cusp --- and (iii) slow decay toward a constant for the rest of the over-parametrized regime. Thereby we identify the region where the classical phenomenon of over-fitting takes place, and the region where the model keeps improving, in line with previous empirical observations for modern neural networks.

arXiv commentarXiv admin note: text overlap with arXiv:1809.09349

References in corpus (8)

• Theoretical perspective on the glass transition and amorphous materials
• Effects of compression on the vibrational modes of marginally jammed solids
• On the Rigidity of Amorphous Solids
• Universal Spectrum of Normal Modes in Low-Temperature Glasses: an Exact Solution
• Stress Propagation through Frictionless Granular Material
• The simplest model of jamming
• Perspective: Energy Landscapes for Machine Learning
• Phonon gap and localization lengths in floppy materials

Cited by in corpus (25)

• Scaling description of generalization with number of parameters in deep learning
• Two models of double descent for weak features
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Generalisation error in learning with random features and the hidden manifold model
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• Self-explaining AI as an alternative to interpretable AI
• On the different regimes of Stochastic Gradient Descent
• Learning through atypical "phase transitions" in overparameterized neural networks
• Probing transfer learning with a model of synthetic correlated datasets
• Geometric compression of invariant manifolds in neural nets
• Finite size effects in the microscopic critical properties of jammed configurations: A comprehensive study of the effects of different types of disorder
• Fitting very flexible models: Linear regression with large numbers of parameters
• Eight challenges in developing theory of intelligence
• Hard-Sphere Jamming through the Lens of Linear Optimization
• Dimensionality reduction, regularization, and generalization in overparameterized regressions
• Data-driven effective model shows a liquid-like deep learning
• Can we avoid Double Descent in Deep Neural Networks?
• Subaging in underparametrized Deep Neural Networks
• The geometry of jamming algorithms in the random Lorentz gas
• Double descent in quantum kernel methods
• The Double Descent Behavior in Two Layer Neural Network for Binary Classification
• Estimating the mass of galactic components using machine learning algorithms
• Gradient descent dynamics and the jamming transition in infinite dimensions
• Tradeoff of generalization error in unsupervised learning