AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations

Hongwei Zhang, Weidong Zou, Hongbo Zhao +4

Adaptive optimization methods have been widely used in deep learning. They scale the learning rates adaptively according to the past gradient, which has been shown to be effective…