Rapidly Adapting Moment Estimation

Abstract

Adaptive gradient methods such as Adam have been shown to be very effective for training deep neural networks (DNNs) by tracking the second moment of gradients to compute the individual learning rates. Differently from existing methods, we make use of the most recent first moment of gradients to compute the individual learning rates per iteration. The motivation behind it is that the dynamic variation of the first moment of gradients may provide useful information to obtain the learning rates. We refer to the new method as the rapidly adapting moment estimation (RAME). The theoretical convergence of deterministic RAME is studied by using an analysis similar to the one used in [1] for Adam. Experimental results for training a number of DNNs show promising performance of RAME w.r.t. the convergence speed and generalization performance compared to the stochastic heavy-ball (SHB) method, Adam, and RMSprop.

arXiv comment11 pages

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• On the Convergence of Adam and Beyond
• Feature Pyramid Networks for Object Detection
• On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length