On the Convergence of Vanilla Adam under Heavy-Tailed Noise and Index Sampling

Kanata Oowada, Hideaki Iiduka

Understanding how stochastic gradient noise affects the convergence of Adam is an important problem in stochastic optimization, particularly when the noise exhibits heavy-tailed be…