On the Convergence Proof of AMSGrad and a New Version

Tran Thi Phuong, Le Trieu Phong

The adaptive moment estimation algorithm Adam (Kingma and Ba) is a popular optimizer in the training of deep neural networks. However, Reddi et al. have recently shown that the con…