AdaX: Adaptive Gradient Descent with Exponential Long Term Memory
arXiv:2004.09740
Abstract
Although adaptive optimization algorithms such as Adam show fast convergence in many machine learning tasks, this paper identifies a problem of Adam by analyzing its performance in a simple non-convex synthetic problem, showing that Adam's fast convergence would possibly lead the algorithm to local minimums. To address this problem, we improve Adam by proposing a novel adaptive gradient descent algorithm named AdaX. Unlike Adam that ignores the past gradients, AdaX exponentially accumulates the long-term gradient information in the past during training, to adaptively tune the learning rate. We thoroughly prove the convergence of AdaX in both the convex and non-convex settings. Extensive experiments show that AdaX outperforms Adam in various tasks of computer vision and natural language processing and can catch up with Stochastic Gradient Descent.
References in corpus (11)
- Decoupled Weight Decay Regularization
- ADADELTA: An Adaptive Learning Rate Method
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- SGDR: Stochastic Gradient Descent with Warm Restarts
- On the Convergence of Adam and Beyond
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- Adaptive Bound Optimization for Online Convex Optimization
- Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
- On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
- Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
Cited by in corpus (5)
- AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
- Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
- Context-aware Cross-level Fusion Network for Camouflaged Object Detection
- AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations
- Momentum Centering and Asynchronous Update for Adaptive Gradient Methods