Can SGD Handle Heavy-Tailed Noise?

Ilyas Fatkhullin, Florian Hübler, Guanghui Lan

Stochastic Gradient Descent (SGD) is a cornerstone of large-scale optimization, yet its theoretical behavior under heavy-tailed noise -- common in modern machine learning and reinf…