Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers

Annalisa Belloni, Lorenzo Noci, Antonio Orvieto

The Warmup Stable Decay (WSD) learning rate scheduler has recently become popular, largely due to its good performance and flexibility when training large language models. It remai…