On the Trend-corrected Variant of Adaptive Stochastic Optimization Methods
arXiv:2001.06130 · doi:10.1109/IJCNN48605.2020.9207166
Abstract
Adam-type optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning. Such methods are appealing due to the capability on large-scale sparse datasets with high computational efficiency. In this paper, we present a new framework for Adam-type methods with the trend information when updating the parameters with the adaptive step size and gradients. The additional terms in the algorithm promise an efficient movement on the complex cost surface, and thus the loss would converge more rapidly. We show empirically the importance of adding the trend component, where our framework outperforms the conventional Adam and AMSGrad methods constantly on the classical models with several real-world datasets.
arXiv comment8 pages, 4 figures, 2 tables, IJCNN2020
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Decoupled Weight Decay Regularization
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- ADADELTA: An Adaptive Learning Rate Method
- An overview of gradient descent optimization algorithms
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- On the Convergence of Adam and Beyond
- On the Variance of the Adaptive Learning Rate and Beyond
- On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
- Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
- On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization