A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

Abstract

The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kicks in. This assumption is often made for mathematical convenience, since it enables SGD to be analyzed as a stochastic differential equation (SDE) driven by a Brownian motion. We argue that the Gaussianity assumption might fail to hold in deep learning settings and hence render the Brownian motion-based analyses inappropriate. Inspired by non-Gaussian natural phenomena, we consider the GN in a more general context and invoke the generalized CLT (GCLT), which suggests that the GN converges to a heavy-tailed $\alpha$-stable random variable. Accordingly, we propose to analyze SGD as an SDE driven by a Lévy motion. Such SDEs can incur `jumps', which force the SDE transition from narrow minima to wider minima, as proven by existing metastability theory. To validate the $\alpha$-stable assumption, we conduct extensive experiments on common deep learning architectures and show that in all settings, the GN is highly non-Gaussian and admits heavy-tails. We further investigate the tail behavior in varying network architectures and sizes, loss functions, and datasets. Our results open up a different perspective and shed more light on the belief that SGD prefers wide minima.

Cited by in corpus (9)

• Non-Gaussianity of Stochastic Gradient Noise
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• A Study of Gradient Variance in Deep Learning
• First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
• High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics
• Escaping Saddle Points with Stochastically Controlled Stochastic Gradient Methods
• Training Efficiency and Robustness in Deep Learning