Showing 2024Show all
2 papers · 1 filter
cs.LG2024
Convergence Rate Analysis of LION
Yiming Dong, Huan Li, Zhouchen Lin
The LION (evoLved sIgn mOmeNtum) optimizer for deep neural network training was found by Google via program search, with the simple sign update yet showing impressive performance i…
math.OC2024
On the Convergence Rate of RMSProp and Its Momentum Extension Measured by Norm
Huan Li, Yiming Dong, Zhouchen Lin
Although adaptive gradient methods have been extensively used in deep learning, their convergence rates proved in the literature are all slower than that of SGD, particularly with…