AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods

Abstract

Adam is shown not being able to converge to the optimal solution in certain cases. Researchers recently propose several algorithms to avoid the issue of non-convergence of Adam, but their efficiency turns out to be unsatisfactory in practice. In this paper, we provide new insight into the non-convergence issue of Adam as well as other adaptive learning rate methods. We argue that there exists an inappropriate correlation between gradient $g_t$ and the second-moment term $v_t$ in Adam ($t$ is the timestep), which results in that a large gradient is likely to have small step size while a small gradient may have a large step size. We demonstrate that such biased step sizes are the fundamental cause of non-convergence of Adam, and we further prove that decorrelating $v_t$ and $g_t$ will lead to unbiased step size for each gradient, thus solving the non-convergence problem of Adam. Finally, we propose AdaShift, a novel adaptive learning rate method that decorrelates $v_t$ and $g_t$ by temporal shifting, i.e., using temporally shifted gradient $g_{t-n}$ to calculate $v_t$. The experiment results demonstrate that AdaShift is able to address the non-convergence issue of Adam, while still maintaining a competitive performance with Adam in terms of both training speed and generalization.

arXiv commentPublished as a conference paper at ICLR 2019

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• Improving Generalization Performance by Switching from Adam to SGD

Cited by in corpus (13)

• On Empirical Comparisons of Optimizers for Deep Learning
• A survey of synthetic data augmentation methods in computer vision
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• A Sufficient Condition for Convergences of Adam and RMSProp
• Does Adam optimizer keep close to the optimal point?
• Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
• Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
• Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs
• AdaSGD: Bridging the gap between SGD and Adam
• Follow Your Path: a Progressive Method for Knowledge Distillation
• Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning
• Adaptive Gradient Method with Resilience and Momentum