Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler

Yikang Shen, Matthew Stallone, Mayank Mishra +6

Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch siz…