Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method

Aaron Defazio, Baoyu Zhou, Lin Xiao

The classical AdaGrad method adapts the learning rate by dividing by the square root of a sum of squared gradients. Because this sum on the denominator is increasing, the method ca…