Why Warmup the Learning Rate? Underlying Mechanisms and Improvements

Dayal Singh Kalra, Maissam Barkeshli

It is common in deep learning to warm up the learning rate $\eta$, often by a linear schedule between ${\eta }_{\text{init}}=0$ and a predetermined target ${\eta }_{\text{trgt}}$. In this paper…