13 citations · 30 across the 5 of their papers we have counts for
5 papers
Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case
Meixuan He, Yuqing Liang, Jinlan Liu +1
Adam is a commonly used stochastic optimization algorithm in machine learning. However, its convergence is still not fully understood, especially in the non-convex setting. This pa…
UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization
Yiming Jiang, Jinlan Liu, Dongpo Xu +1
Adam-type algorithms have become a preferred choice for optimisation in the deep learning setting, however, despite success, their convergence is still not well understood. To this…
Last-iterate convergence analysis of stochastic momentum methods for neural networks
Dongpo Xu, Jinlan Liu, Yinghua Lu +2
The stochastic momentum method is a commonly used acceleration technique for solving large-scale stochastic optimization problems in artificial neural networks. Current convergence…
Scaling transition from momentum stochastic gradient descent to plain stochastic gradient descent
Kun Zeng, Jinlan Liu, Zhixia Jiang +1
The plain stochastic gradient descent and momentum stochastic gradient descent have extremely wide applications in deep learning due to their simple settings and low computational…
A decreasing scaling transition scheme from Adam to SGD
Kun Zeng, Jinlan Liu, Zhixia Jiang +1
Adaptive gradient algorithm (AdaGrad) and its variants, such as RMSProp, Adam, AMSGrad, etc, have been widely used in deep learning. Although these algorithms are faster in the ear…