most citedRandom-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers

4 citations · 6 across the 3 of their papers we have counts for

collaborators
Showing stat.MLShow all

1 paper · 1 filter

stat.ML2021

AdaLoss: A computationally-efficient and provably convergent adaptive gradient method

Xiaoxia Wu, Yuege Xie, Simon Du +1

We propose a computationally-friendly adaptive learning rate schedule, "AdaLoss", which directly uses the information of the loss function to adjust the stepsize in gradient descen…