1 citations · 1 across the 9 of their papers we have counts for
Showing 2026Show all
3 papers · 1 filter
cs.LG2026
How Does Local Landscape Geometry Evolve in Language Model Pre-Training?
Zhanpeng Zhou, Yuhan Sun, Bingrui Li +4
The scale and expense of pre-training language models make efficient hyperparameter tuning essential, yet a principled guidance is still missing. In this work, we analyze language…
stat.ML2026
Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency
Jia-Nan Wang, Zixun Huang, Kairui Li +2
We study when and how momentum improves large-batch training in the one-pass regime, using power-law kernel regression as a tractable setting. We first characterize stochastic stab…
cs.LG2026
Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining
Zihan Liu, Ruiheng Zheng, Shaobo Zhang +4
We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR).…