A New Adaptive Gradient Method with Gradient Decomposition

Zhou Shao, Tong Lin

Adaptive gradient methods, especially Adam-type methods (such as Adam, AMSGrad, and AdaBound), have been proposed to speed up the training process with an element-wise scaling term…