Adaptive Gradient Method with Resilience and Momentum

Abstract

Several variants of stochastic gradient descent (SGD) have been proposed to improve the learning effectiveness and efficiency when training deep neural networks, among which some recent influential attempts would like to adaptively control the parameter-wise learning rate (e.g., Adam and RMSProp). Although they show a large improvement in convergence speed, most adaptive learning rate methods suffer from compromised generalization compared with SGD. In this paper, we proposed an Adaptive Gradient Method with Resilience and Momentum (AdaRem), motivated by the observation that the oscillations of network parameters slow the training, and give a theoretical proof of convergence. For each parameter, AdaRem adjusts the parameter-wise learning rate according to whether the direction of one parameter changes in the past is aligned with the direction of the current gradient, and thus encourages long-term consistent parameter updating with much fewer oscillations. Comprehensive experiments have been conducted to verify the effectiveness of AdaRem when training various models on a large-scale image recognition dataset, e.g., ImageNet, which also demonstrate that our method outperforms previous adaptive learning rate-based algorithms in terms of the training speed and the test error, respectively.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• Regularizing and Optimizing LSTM Language Models
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• An Exponential Learning Rate Schedule for Deep Learning
• AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
• Revisiting the Sibling Head in Object Detector