On the different regimes of Stochastic Gradient Descent

Abstract

Modern deep networks are trained with stochastic gradient descent (SGD) whose key hyperparameters are the number of data considered at each step or batch size $B$, and the step size or learning rate $\eta$. For small $B$ and large $\eta$, SGD corresponds to a stochastic evolution of the parameters, whose noise amplitude is governed by the ''temperature'' $T\equiv \eta /B$. Yet this description is observed to break down for sufficiently large batches $B\ge B^{\ast }$, or simplifies to gradient descent (GD) when the temperature is sufficiently small. Understanding where these cross-overs take place remains a central challenge. Here, we resolve these questions for a teacher-student perceptron classification model and show empirically that our key predictions still apply to deep networks. Specifically, we obtain a phase diagram in the $B$-$\eta$ plane that separates three dynamical phases: (i) a noise-dominated SGD governed by temperature, (ii) a large-first-step-dominated SGD and (iii) GD. These different phases also correspond to different regimes of generalization error. Remarkably, our analysis reveals that the batch size $B^{\ast }$ separating regimes (i) and (ii) scale with the size $P$ of the training set, with an exponent that characterizes the hardness of the classification problem.

Main: 8 pages, 4 figures; Appendix: 15 pages, 11 figures

References in corpus (9)

• Exploring Generalization in Deep Learning
• How to Escape Saddle Points Efficiently
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks
• SGD with Large Step Sizes Learns Sparse Features
• Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning

Cited by in corpus (2)

• Stochastic weight matrix dynamics during learning and Dyson Brownian motion
• Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks