Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

Zihan Liu, Ruiheng Zheng, Shaobo Zhang +4

We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR).…