Non-Convergence and Limit Cycles in the Adam optimizer

Sebastian Bock, Martin Georg Weiß

One of the most popular training algorithms for deep neural networks is the Adaptive Moment Estimation (Adam) introduced by Kingma and Ba. Despite its success in many applications…