Jamming in multilayer supervised learning models

Abstract

Critical jamming transitions are characterized by an astonishing degree of universality. Analytic and numerical evidence points to the existence of a large universality class that encompasses finite and infinite dimensional spheres and continuous constraint satisfaction problems (CCSP) such as the non-convex perceptron and related models. In this paper we investigate multilayer neural networks (MLNN) learning random associations as models for CCSP which could potentially define different jamming universality classes. As opposed to simple perceptrons and infinite dimensional spheres, which are described by a single effective field in terms of which the constraints appear to be one-dimensional, the description of MLNN, involves multiple fields, and the constraints acquire a multidimensional character. We first study the models numerically and show that similarly to the perceptron, whenever jamming is isostatic, the sphere universality class is recovered, we then write the exact mean-field equations for the models and identify a dimensional reduction mechanism that leads to a scaling regime identical to one of the infinite dimensional spheres. We suggest that this mechanism could be general enough to explain finite dimensional universality.

arXiv comment5+12 pages

References in corpus (19)

• Jamming at Zero Temperature and Zero Applied Stress: the Epitome of Disorder
• Mean field theory of hard sphere glasses and jamming
• Fractal free energy landscapes in structural glasses
• Statistical physics of inference: Thresholds and algorithms
• Effects of compression on the vibrational modes of marginally jammed solids
• Glass and Jamming Transitions: From Exact Results to Finite-Dimensional Descriptions
• Marginal Stability in Structural, Spin and Electron Glasses
• Universal Spectrum of Normal Modes in Low-Temperature Glasses: an Exact Solution
• Stress Propagation through Frictionless Granular Material
• Exact theory of dense amorphous hard spheres in high dimension. III. The full RSB solution
• Stability at Random Close Packing
• Jamming Criticality Revealed by Removing Localized Buckling Excitations
• The simplest model of jamming
• Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Universality of jamming of non-spherical particles
• Glassy nature of the hard phase in inference problems
• Disorder-free spin glass transitions and jamming in exactly solvable mean-field models
• Mean field theory of jamming of nonspherical particles

Cited by in corpus (23)

• Scaling description of generalization with number of parameters in deep learning
• Optimization for deep learning: theory and algorithms
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Perspective: Gardner Physics in Amorphous Solids and Beyond
• The effective noise of Stochastic Gradient Descent
• Jamming with tunable roughness
• Finite size effects in the microscopic critical properties of jammed configurations: A comprehensive study of the effects of different types of disorder
• Critical energy landscape of linear soft spheres
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• High dimensional optimization under non-convex excluded volume constraints
• Surfing on minima of isostatic landscapes: avalanches and unjamming transition
• Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks
• Tractability from overparametrization: The example of the negative perceptron
• Dynamical mean field theory for models of confluent tissues and beyond
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• Proliferation of non-linear excitations in the piecewise-linear perceptron
• Non-Affine Displacements Below Jamming under Athermal Quasi-Static Compression
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Localization of quantum walk with classical randomness: Comparison between manual methods and supervised machine learning
• Satisfiability transition in asymmetric neural networks
• How isotropic kernels perform on simple invariants
• Critical properties of the SAT/UNSAT transitions in the classification problem of structured data
• Gradient descent dynamics and the jamming transition in infinite dimensions