Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
arXiv:2006.06098 · doi:10.1088/1742-5468/ac3a80
Abstract
We analyze in a closed form the learning dynamics of stochastic gradient descent (SGD) for a single-layer neural network classifying a high-dimensional Gaussian mixture where each cluster is assigned one of two labels. This problem provides a prototype of a non-convex loss landscape with interpolating regimes and a large generalization gap. We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow. In the full-batch limit, we recover the standard gradient flow. We apply dynamical mean-field theory from statistical physics to track the dynamics of the algorithm in the high-dimensional limit via a self-consistent stochastic process. We explore the performance of the algorithm as a function of the control parameters shedding light on how it navigates the loss landscape.
arXiv comment8 pages + appendix, 4 figures
References in corpus (15)
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Exponential expressivity in deep neural networks through transient chaos
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
- Numerical implementation of dynamical mean field theory for disordered systems: application to the Lotka-Volterra model of ecosystems
- Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
- A Mean Field Theory of Batch Normalization
- Out-of-equilibrium dynamical mean-field equations for the perceptron model
- Jamming in multilayer supervised learning models
- Mean-field inference methods for neural networks
- Numerical solution of the dynamical mean field theory of infinite-dimensional equilibrium liquids
- Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference
- Bad Global Minima Exist and SGD Can Reach Them
- Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs