Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM

Qianqian Tong, Guannan Liang, Jinbo Bi

Adaptive gradient methods (AGMs) have become popular in optimizing the nonconvex problems in deep learning area. We revisit AGMs and identify that the adaptive learning rate (A-LR)…