AdaLoss: A computationally-efficient and provably convergent adaptive gradient method

Xiaoxia Wu, Yuege Xie, Simon Du +1

We propose a computationally-friendly adaptive learning rate schedule, "AdaLoss", which directly uses the information of the loss function to adjust the stepsize in gradient descen…