Out-of-equilibrium dynamical mean-field equations for the perceptron model

Abstract

Perceptrons are the building blocks of many theoretical approaches to a wide range of complex systems, ranging from neural networks and deep learning machines, to constraint satisfaction problems, glasses and ecosystems. Despite their applicability and importance, a detailed study of their Langevin dynamics has never been performed yet. Here we derive the mean-field dynamical equations that describe the continuous random perceptron in the thermodynamic limit, in a very general setting with arbitrary noise and friction kernels, not necessarily related by equilibrium relations. We derive the equations in two ways: via a dynamical cavity method, and via a path-integral approach in its supersymmetric formulation. The end point of both approaches is the reduction of the dynamics of the system to an effective stochastic process for a representative dynamical variable. Because the perceptron is formally very close to a system of interacting particles in a high dimensional space, the methods we develop here can be transferred to the study of liquid and glasses in high dimensions. Potentially interesting applications are thus the study of the glass transition in active matter, the study of the dynamics around the jamming transition, and the calculation of rheological properties in driven systems.

References in corpus (14)

• Theoretical perspective on the glass transition and amorphous materials
• Supercooled Liquids for Pedestrians
• Gibbs States and the Set of Solutions of Random Constraint Satisfaction Problems
• Fractal free energy landscapes in structural glasses
• Glass Rheology: From mode-coupling theory to a dynamical yield criterion
• A collective phase in resource competition in a highly diverse ecosystem
• Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
• Unified Theory of Inertial Granular Flows and Non-Brownian Suspensions
• Generalization of the cavity method for adiabatic evolution of Gibbs states
• Diverging viscosity and soft granular rheology in non-Brownian suspensions
• Mean-field avalanches in jammed spheres
• A simple theory for the dynamics of mean-field-like models of glass-forming fluids
• The jamming transition in high dimension: an analytical study of the TAP equations and the effective thermodynamic potential
• Non equilibrium dynamics of isolated disordered systems: the classical Hamiltonian p-spin model

Cited by in corpus (33)

• Dimension of activity in random neural networks
• The effective noise of Stochastic Gradient Descent
• Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
• Numerical solution of the dynamical mean field theory of infinite-dimensional equilibrium liquids
• Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference
• Out-of-equilibrium dynamical equations of infinite-dimensional particle systems. I. The isotropic case
• Memory formation in jammed hard spheres
• On the Dynamics of Liquids in the Large-Dimensional Limit
• On weak ergodicity breaking in mean-field spin glasses
• Mean-field caging in a random Lorentz gas
• Dynamical Mean-Field Theory and Aging Dynamics
• Aging by near-extinctions in many-variable interacting populations
• Out-of-equilibrium dynamical equations of infinite-dimensional particle systems. II. The anisotropic case under shear strain
• Impact of jamming criticality on low-temperature anomalies in structural glasses
• Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings
• A continuous constraint satisfaction problem for the rigidity transition in confluent tissues
• Rigorous dynamical mean field theory for stochastic gradient descent methods
• Introduction to the dynamics of disordered systems: equilibrium and gradient descent
• Surfing on minima of isostatic landscapes: avalanches and unjamming transition
• Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
• Force balance controls the relaxation time of the gradient descent algorithm in the satisfiable phase
• Jamming, relaxation, and memory in a structureless glass former
• Dynamical mean field theory for models of confluent tissues and beyond
• Tractability from overparametrization: The example of the negative perceptron
• Thresholds of descending algorithms in inference problems
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Models
• Dynamics and fluctuations of minimally-structured glass formers
• Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems
• Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model
• Comment on "Explicit Analytical Solution for Random Close Packing in $d=2$ and $d=3$"
• Simplified derivations for high-dimensional convex learning problems
• Gradient descent dynamics and the jamming transition in infinite dimensions