Divergence Results and Convergence of a Variance Reduced Version of ADAM

Ruiqi Wang, Diego Klabjan

Stochastic optimization algorithms using exponential moving averages of the past gradients, such as ADAM, RMSProp and AdaGrad, have been having great successes in many applications…