Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descent

Krunoslav Lehman Pavasovic, Alain Durmus, Umut Simsekli

A recent line of empirical studies has demonstrated that SGD might exhibit a heavy-tailed behavior in practical settings, and the heaviness of the tails might correlate with the ov…