The effective noise of Stochastic Gradient Descent
arXiv:2112.10852 · doi:10.1088/1742-5468/ac841d
Abstract
Stochastic Gradient Descent (SGD) is the workhorse algorithm of deep learning technology. At each step of the training phase, a mini batch of samples is drawn from the training dataset and the weights of the neural network are adjusted according to the performance on this specific subset of examples. The mini-batch sampling procedure introduces a stochastic dynamics to the gradient descent, with a non-trivial state-dependent noise. We characterize the stochasticity of SGD and a recently-introduced variant, \emph{persistent} SGD, in a prototypical neural network model. In the under-parametrized regime, where the final training error is positive, the SGD dynamics reaches a stationary state and we define an effective temperature from the fluctuation-dissipation theorem, computed from dynamical mean-field theory. We use the effective temperature to quantify the magnitude of the SGD noise as a function of the problem parameters. In the over-parametrized regime, where the training error vanishes, we measure the noise magnitude of SGD by computing the average distance between two replicas of the system with the same initialization and two different realizations of SGD noise. We find that the two noise measures behave similarly as a function of the problem parameters. Moreover, we observe that noisier algorithms lead to wider decision boundaries of the corresponding constraint satisfaction problem.
arXiv comment7 pages + appendix, 5 figures
References in corpus (13)
- Analytical Solution of the Off-Equilibrium Dynamics of a Long Range Spin-Glass Model
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Nonequilibrium glass transitions in driven and active matter
- The effective temperature
- Effective temperature of active matter
- Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
- Path Integral Approach to Random Neural Networks
- Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning
- Out-of-equilibrium dynamical mean-field equations for the perceptron model
- Jamming in multilayer supervised learning models
- Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
- How to Study a Persistent Active Glassy System?
- Force balance controls the relaxation time of the gradient descent algorithm in the satisfiable phase