Progressive Residual Warmup for Language Model Pretraining

Tianhao Chen, Xin Xu, Lu Yin +4

Transformer architectures serve as the backbone for most modern Large Language Models, therefore their pretraining stability and convergence speed are of central concern. Motivated…