DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization

Abstract

Optimization algorithms with momentum, e.g., (ADAM), have been widely used for building deep learning models due to the faster convergence rates compared with stochastic gradient descent (SGD). Momentum helps accelerate SGD in the relevant directions in parameter updating, which can minify the oscillations of parameters update route. However, there exist errors in some update steps in optimization algorithms with momentum like ADAM. The fixed momentum weight (e.g., β_1 in ADAM) will propagate errors in momentum computing. In this paper, we introduce a novel optimization algorithm, namely Discriminative wEight on Adaptive Momentum (DEAM). Instead of assigning the momentum term weight with a fixed hyperparameter, DEAM proposes to compute the momentum weight automatically based on the discriminative angle. In this way, DEAM involves fewer hyperparameters. DEAM also contains a novel backtrack term, which restricts redundant updates when the correction of the last step is needed. Extensive experiments demonstrate that DEAM can achieve a faster convergence rate than the existing optimization algorithms in training the deep learning models of both convex and non-convex situations.

References in corpus (10)

• ADADELTA: An Adaptive Learning Rate Method
• An overview of gradient descent optimization algorithms
• On the Convergence of Adam and Beyond
• Improving Generalization Performance by Switching from Adam to SGD
• Stochastic Variance Reduction for Nonconvex Optimization
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Equilibrated adaptive learning rates for non-convex optimization
• Convergence Analysis of Proximal Gradient with Momentum for Nonconvex Optimization
• Normalized Direction-preserving Adam
• Graph Neural Networks for Small Graph and Giant Network Representation Learning: An Overview