Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates

Steffen Dereich, Robin Graeber, Arnulf Jentzen

Deep learning algorithms - typically consisting of a class of deep neural networks trained by a stochastic gradient descent (SGD) optimization method - are nowadays the key ingredi…