MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients

Chen Zhu, Yu Cheng, Zhe Gan +3

Adaptive gradient methods such as RMSProp and Adam use exponential moving estimate of the squared gradient to compute adaptive step sizes, achieving better convergence than SGD in…