On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks

Umut Şimşekli, Mert Gürbüzbalaban, Thanh Huy Nguyen +2

The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the \emph{classical} central…