3 papers
cs.LG2024
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
Bohan Wang, Huishuai Zhang, Qi Meng +3
This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates. We demonstrate that Adam achieves a fa…
cs.LG2023
Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity
Bohan Wang, Jingwen Fu, Huishuai Zhang +2
Recently, Arjevani et al. [1] established a lower bound of iteration complexity for the first-order optimization under an -smooth condition and a bounded noise variance assumpti…
cs.DS2020
Constructing Basis Path Set by Eliminating Path Dependency
Juanping Zhu, Qi Meng, Wei Chen +2
The way the basis path set works in neural network remains mysterious, and the generalization of newly appeared G-SGD algorithm to more practical network is hindered. The Basis Pat…