Comparing Dynamics: Deep Neural Networks versus Glassy Systems

Abstract

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and datasets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.

arXiv comment10 pages, 5 figures. Version accepted at ICML 2018

References in corpus (12)

• Theoretical perspective on the glass transition and amorphous materials
• Analytical Solution of the Off-Equilibrium Dynamics of a Long Range Spin-Glass Model
• Gibbs States and the Set of Solutions of Random Constraint Satisfaction Problems
• Spin-Glass Theory for Pedestrians
• Statistical physics of inference: Thresholds and algorithms
• Models and algorithms for the next generation of glass transition studies
• On the Rigidity of Amorphous Solids
• Rigorous Inequalities between Length and Time Scales in Glassy Systems
• Phase space geometry and slow dynamics
• Theory for Swap Acceleration near the Glass and Jamming Transitions
• Spectral gap estimates in mean field spin glasses
• Activated Aging Dynamics and Effective Trap Model Description in the Random Energy Model

Cited by in corpus (20)

• Machine Learning for Condensed Matter Physics
• Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach
• Relaxation dynamics in the energy landscape of glass-forming liquids
• Memory formation in jammed hard spheres
• On weak ergodicity breaking in mean-field spin glasses
• Mean-field dynamics of infinite-dimensional particle systems: global shear versus random local forcing
• The Loss Surfaces of Neural Networks with General Activation Functions
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• Nonreciprocal Spin-Glass Transition and Aging
• A spin-glass model for the loss surfaces of generative adversarial networks
• Numerical renormalization of glassy dynamics
• On-Site Potential Creates Complexity in Systems with Disordered Coupling
• Thermodynamics of the Ising model encoded in restricted Boltzmann machines
• Black holes and the loss landscape in machine learning
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Data-driven effective model shows a liquid-like deep learning
• An analytic theory of shallow networks dynamics for hinge loss classification
• Subaging in underparametrized Deep Neural Networks
• Tradeoff of generalization error in unsupervised learning
• Gradient descent dynamics and the jamming transition in infinite dimensions