Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics

Abstract

In this paper, we characterize the noise of stochastic gradients and analyze the noise-induced dynamics during training deep neural networks by gradient-based optimizers. Specifically, we firstly show that the stochastic gradient noise possesses finite variance, and therefore the classical Central Limit Theorem (CLT) applies; this indicates that the gradient noise is asymptotically Gaussian. Such an asymptotic result validates the wide-accepted assumption of Gaussian noise. We clarify that the recently observed phenomenon of heavy tails within gradient noise may not be intrinsic properties, but the consequence of insufficient mini-batch size; the gradient noise, which is a sum of limited i.i.d. random variables, has not reached the asymptotic regime of CLT, thus deviates from Gaussian. We quantitatively measure the goodness of Gaussian approximation of the noise, which supports our conclusion. Secondly, we analyze the noise-induced dynamics of stochastic gradient descent using the Langevin equation, granting for momentum hyperparameter in the optimizer with a physical interpretation. We then proceed to demonstrate the existence of the steady-state distribution of stochastic gradient descent and approximate the distribution at a small learning rate.

arXiv comment18 pages

References in corpus (13)

• A Survey of Deep Learning Techniques for Autonomous Driving
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Game-Theoretic Multiagent Reinforcement Learning
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• On the Noisy Gradient Descent that Generalizes as SGD
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Non-Gaussianity of Stochastic Gradient Noise
• Towards Understanding the Importance of Noise in Training Neural Networks
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Thermostat-assisted continuously-tempered Hamiltonian Monte Carlo for Bayesian learning
• Benchmarking Deep Sequential Models on Volatility Predictions for Financial Time Series