Scaling Law with Learning Rate Annealing

Howe Tissue, Venus Wang, Lu Wang

We find that the cross-entropy loss curves of neural language models empirically adhere to a scaling law with learning rate (LR) annealing over training steps: $$L(s) = L_0 + A\cdo…