136 citations · 436 across the 9 of their papers we have counts for
9 papers
Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case
Meixuan He, Yuqing Liang, Jinlan Liu +1
Adam is a commonly used stochastic optimization algorithm in machine learning. However, its convergence is still not fully understood, especially in the non-convex setting. This pa…
Convex Quaternion Optimization for Signal Processing: Theory and Applications
Shuning Sun, Qiankun Diao, Dongpo Xu +2
Convex optimization methods have been extensively used in the fields of communications and signal processing. However, the theory of quaternion optimization is currently not as ful…
UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization
Yiming Jiang, Jinlan Liu, Dongpo Xu +1
Adam-type algorithms have become a preferred choice for optimisation in the deep learning setting, however, despite success, their convergence is still not well understood. To this…
Last-iterate convergence analysis of stochastic momentum methods for neural networks
Dongpo Xu, Jinlan Liu, Yinghua Lu +2
The stochastic momentum method is a commonly used acceleration technique for solving large-scale stochastic optimization problems in artificial neural networks. Current convergence…
Scaling transition from momentum stochastic gradient descent to plain stochastic gradient descent
Kun Zeng, Jinlan Liu, Zhixia Jiang +1
The plain stochastic gradient descent and momentum stochastic gradient descent have extremely wide applications in deep learning due to their simple settings and low computational…
A decreasing scaling transition scheme from Adam to SGD
Kun Zeng, Jinlan Liu, Zhixia Jiang +1
Adaptive gradient algorithm (AdaGrad) and its variants, such as RMSProp, Adam, AMSGrad, etc, have been widely used in deep learning. Although these algorithms are faster in the ear…
The Theory of Quaternion Matrix Derivatives
Dongpo Xu, Danilo P. Mandic
A systematic theory is introduced for calculating the derivatives of quaternion matrix function with respect to quaternion matrix variables. The proposed methodology is equipped wi…
Quaternion Derivatives: The GHR Calculus
Dongpo Xu, Cyrus Jahanchahi, Clive C. Took +1
Quaternion derivatives in the mathematical literature are typically defined only for analytic (regular) functions. However, in engineering problems, functions of interest are often…
Quaternion Gradient and Hessian
Dongpo Xu, Danilo P. Mandic
The optimization of real scalar functions of quaternion variables, such as the mean square error or array output power, underpins many practical applications. Solutions often requi…