On the Nonlinearity of Learning Rate Scaling for LLM Training

Zaiwen Yang, Huaqing Zhang, Jing Xu +1

Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Exist…