On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond

Bohan Wang, Huishuai Zhang, Qi Meng +3

This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates. We demonstrate that Adam achieves a fa…