Proliferation of non-linear excitations in the piecewise-linear perceptron

Abstract

We investigate the properties of local minima of the energy landscape of a continuous non-convex optimization problem, the spherical perceptron with piecewise linear cost function and show that they are critical, marginally stable and displaying a set of pseudogaps, singularities and non-linear excitations whose properties appear to be in the same universality class of jammed packings of hard spheres. The piecewise linear perceptron problem appears as an evolution of the purely linear perceptron optimization problem that has been recently investigated in [1]. Its cost function contains two non-analytic points where the derivative has a jump. Correspondingly, in the non-convex/glassy phase, these two points give rise to four pseudogaps in the force distribution and this induces four power laws in the gap distribution as well. In addition one can define an extended notion of isostaticity and show that local minima appear again to be isostatic in this phase. We believe that our results generalize naturally to more complex cases with a proliferation of non-linear excitations as the number of non-analytic points in the cost function is increased.

arXiv comment14 pages, 7 figures

References in corpus (18)

• Fractal free energy landscapes in structural glasses
• Glass and Jamming Transitions: From Exact Results to Finite-Dimensional Descriptions
• Finite-Size Scaling at the Jamming Transition
• Marginal Stability in Structural, Spin and Electron Glasses
• Universal Spectrum of Normal Modes in Low-Temperature Glasses: an Exact Solution
• Exact theory of dense amorphous hard spheres in high dimension. III. The full RSB solution
• Exact theory of dense amorphous hard spheres in high dimension. II. The high density regime and the Gardner transition
• The simplest model of jamming
• Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Two diverging length scales in the structure of jammed packings
• Jamming in multilayer supervised learning models
• Critical jammed phase of the linear perceptron
• Can a large packing be assembled from smaller ones?
• Approximate Survey Propagation for Statistical Inference
• Disorder-free spin glass transitions and jamming in exactly solvable mean-field models
• Critical energy landscape of linear soft spheres
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks

Cited by in corpus (4)

• Marginal stability of soft anharmonic mean field spin glasses
• A continuous constraint satisfaction problem for the rigidity transition in confluent tissues
• Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
• High dimensional optimization under non-convex excluded volume constraints