The jamming transition as a paradigm to understand the loss landscape of deep neural networks

Abstract

Deep learning has been immensely successful at a variety of tasks, ranging from classification to AI. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in FC networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime, and puts forward the surprising result that the ability of fully connected networks to fit random data is independent of their depth. Our observations suggests that this independence also holds for real data. We also study a quantity $\Delta$ which characterizes how well ($\Delta <0$) or badly ($\Delta >0$) a datum is learned. At the critical point it is power-law distributed, $P_{+}(\Delta )\sim {\Delta }^{\theta }$ for $\Delta >0$ and $P_{-}(\Delta )\sim (-\Delta {)}^{-\gamma }$ for $\Delta <0$, with $\theta \approx 0.3$ and $\gamma \approx 0.2$. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.

References in corpus (38)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Theoretical perspective on the glass transition and amorphous materials
• Jamming at Zero Temperature and Zero Applied Stress: the Epitome of Disorder
• On the Number of Linear Regions of Deep Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• The Loss Surfaces of Multilayer Networks
• Fractal free energy landscapes in structural glasses
• Effects of compression on the vibrational modes of marginally jammed solids
• Vibrations and diverging length scales near the unjamming transition
• On the Rigidity of Amorphous Solids
• Phase Transitions in the Coloring of Random Graphs
• Marginal Stability in Structural, Spin and Electron Glasses
• Universal Spectrum of Normal Modes in Low-Temperature Glasses: an Exact Solution
• Stress Propagation through Frictionless Granular Material
• Exact theory of dense amorphous hard spheres in high dimension. III. The full RSB solution
• Scaling description of generalization with number of parameters in deep learning
• A Landscape Analysis of Constraint Satisfaction Problems
• Universal microstructure and mechanical stability of jammed packings
• Stability at Random Close Packing
• Jamming Criticality Revealed by Removing Localized Buckling Excitations
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Jamming in Systems Composed of Frictionless Ellipse-Shaped Particles
• The simplest model of jamming
• Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
• Perspective: Energy Landscapes for Machine Learning
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Excitations of Ellipsoid Packings near Jamming
• Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
• Phonon gap and localization lengths in floppy materials
• Universality of jamming of non-spherical particles
• Mean-field avalanches in jammed spheres
• On Variational Arguments for Vibrational Modes near Jamming
• Jamming in multilayer supervised learning models
• On the ability of neural nets to express distributions
• Toward a microscopic description of flow near the jamming threshold
• The loss landscape of overparameterized neural networks
• Estimating the Spectral Density of Large Implicit Matrices
• Stuck in a What? Adventures in Weight Space

Cited by in corpus (61)

• Big-Data Science in Porous Materials: Materials Genomics and Machine Learning
• Scaling description of generalization with number of parameters in deep learning
• Optimization for deep learning: theory and algorithms
• Machine Learning for Condensed Matter Physics
• Self-Adaptive Training: beyond Empirical Risk Minimization
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Perspective: Gardner Physics in Amorphous Solids and Beyond
• Jamming in multilayer supervised learning models
• Optimal Regularization Can Mitigate Double Descent
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Effects of noise on the overparametrization of quantum neural networks
• Mean-field dynamics of infinite-dimensional particle systems: global shear versus random local forcing
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• Finite size effects in the microscopic critical properties of jammed configurations: A comprehensive study of the effects of different types of disorder
• On Power Laws in Deep Ensembles
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
• Asymptotics of Wide Convolutional Neural Networks
• Tuning and jamming reduced to their minima
• Distributional Generalization: A New Kind of Generalization
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• Jamming Criticality of Near-Crystals
• Force balance controls the relaxation time of the gradient descent algorithm in the satisfiable phase
• Fitting very flexible models: Linear regression with large numbers of parameters
• Overparameterized Linear Regression under Adversarial Attacks
• Hard-Sphere Jamming through the Lens of Linear Optimization
• Dimensionality reduction, regularization, and generalization in overparameterized regressions
• Taxonomizing local versus global structure in neural network loss landscapes
• Scaling Down Deep Learning with MNIST-1D
• On the interplay between data structure and loss function in classification problems
• Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
• The asymptotic spectrum of the Hessian of DNN throughout training
• Provable More Data Hurt in High Dimensional Least Squares Estimator
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing
• Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
• Numerical renormalization of glassy dynamics
• Proliferation of non-linear excitations in the piecewise-linear perceptron
• Dynamic Gardner crossover in a simple structural glass
• Thermodynamics of the Ising model encoded in restricted Boltzmann machines
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Data-driven effective model shows a liquid-like deep learning
• Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
• Subaging in underparametrized Deep Neural Networks
• Can we avoid Double Descent in Deep Neural Networks?
• On the Bias-Variance Tradeoff: Textbooks Need an Update
• Double Descent Risk and Volume Saturation Effects: A Geometric Perspective
• A Statistician Teaches Deep Learning
• Family-Vicsek universality of the binary intrinsic dimension of nonequilibrium data
• Double Descent Optimization Pattern and Aliasing: Caveats of Noisy Labels
• Volumization as a Natural Generalization of Weight Decay
• Mixed Semi-Supervised Generalized-Linear-Regression with Applications to Deep-Learning and Interpolators
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective
• Distilling Double Descent