On the different regimes of Stochastic Gradient Descent

Abstract

Modern deep networks are trained with stochastic gradient descent (SGD) whose key hyperparameters are the number of data considered at each step or batch size $B$, and the step size or learning rate $\eta$. For small $B$ and large $\eta$, SGD corresponds to a stochastic evolution of the parameters, whose noise amplitude is governed by the ''temperature'' $T\equiv \eta /B$. Yet this description is observed to break down for sufficiently large batches $B\ge B^{\ast }$, or simplifies to gradient descent (GD) when the temperature is sufficiently small. Understanding where these cross-overs take place remains a central challenge. Here, we resolve these questions for a teacher-student perceptron classification model and show empirically that our key predictions still apply to deep networks. Specifically, we obtain a phase diagram in the $B$-$\eta$ plane that separates three dynamical phases: (i) a noise-dominated SGD governed by temperature, (ii) a large-first-step-dominated SGD and (iii) GD. These different phases also correspond to different regimes of generalization error. Remarkably, our analysis reveals that the batch size $B^{\ast }$ separating regimes (i) and (ii) scale with the size $P$ of the training set, with an exponent that characterizes the hardness of the classification problem.

arXiv commentMain: 8 pages, 4 figures; Appendix: 15 pages, 11 figures

References in corpus (26)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Exploring Generalization in Deep Learning
• How to Escape Saddle Points Efficiently
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• Measuring the Effects of Data Parallelism on Neural Network Training
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Shaping the learning landscape in neural networks around wide flat minima
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Escaping Saddles with Stochastic Gradients
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
• Disentangling feature and lazy training in deep neural networks
• Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations
• Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• A Bayesian Perspective on Generalization and Stochastic Gradient Descent
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• SGD with Large Step Sizes Learns Sparse Features
• Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity
• Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning

Cited by in corpus (2)

• Stochastic weight matrix dynamics during learning and Dyson Brownian motion
• Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks