Strength of Minibatch Noise in SGD
arXiv:2102.05375
Abstract
The noise in stochastic gradient descent (SGD), caused by minibatch sampling, is poorly understood despite its practical importance in deep learning. This work presents the first systematic study of the SGD noise and fluctuations close to a local minimum. We first analyze the SGD noise in linear regression in detail and then derive a general formula for approximating SGD noise in different types of minima. For application, our results (1) provide insight into the stability of training a neural network, (2) suggest that a large learning rate can help generalization by introducing an implicit regularization, (3) explain why the linear learning rate-batchsize scaling law fails at a large learning rate or at a small batchsize and (4) can provide an understanding of how discrete-time nature of SGD affects the recently discovered power-law phenomenon of SGD.
arXiv commentICLR 2022 spotlight
References in corpus (17)
- A Convergence Theory for Deep Learning via Over-Parameterization
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Power Laws are Logarithmic Boltzmann Laws
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
- Theory of Deep Learning IIb: Optimization Properties of SGD
- Fluctuation-dissipation relations for stochastic gradient descent
- Understanding the Role of Momentum in Stochastic Gradient Methods
- On the Noisy Gradient Descent that Generalizes as SGD
- How Good is the Bayes Posterior in Deep Neural Networks Really?
- Multiplicative noise and heavy tails in stochastic optimization
- A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
- A Bayesian Perspective on Generalization and Stochastic Gradient Descent
- Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
- Recent advances in deep learning theory
- Shape Matters: Understanding the Implicit Bias of the Noise Covariance
- Dynamic of Stochastic Gradient Descent with State-Dependent Noise
- On the Distributional Properties of Adaptive Gradients