ADOPT: Modified Adam Can Converge with Any ${\beta }_2$ with the Optimal Rate

Shohei Taniguchi, Keno Harada, Gouki Minegishi +7

Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., ${\beta }_2$,…