Non-Gaussianity of Stochastic Gradient Noise
arXiv:1910.09626
Abstract
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes 256 and above, the distribution is best described as Gaussian at-least in the early phases of training. This holds across data-sets, architectures, and other choices.
References in corpus (5)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Three Factors Influencing Minima in SGD
- Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
- A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
- On the diffusion approximation of nonconvex stochastic gradient descent
Cited by in corpus (13)
- The Heavy-Tail Phenomenon in SGD
- Multiplicative noise and heavy tails in stochastic optimization
- Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
- Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
- On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
- On generalization bounds for deep networks based on loss surface implicit regularization
- Improved generalization by noise enhancement
- An Empirical Study on the Intrinsic Privacy of SGD
- Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
- Understanding Short-Range Memory Effects in Deep Neural Networks
- Fractional moment-preserving initialization schemes for training deep neural networks
- Heavy-Tail Phenomenon in Decentralized SGD
- Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noise