ADOPT: Modified Adam Can Converge with Any ${\hat{I}}_2^2$ with the Optimal Rate

Shohei Taniguchi, Keno Harada, Gouki Minegishi +7

Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., ${\hat{I}}_2^2$,…