Showing 2026Show all
2 papers · 1 filter
cs.LG2026
Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
Shuchen Zhu, Yuxin Fang, Mingze Wang +1
Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditioned loss landscape bring sev…
cs.LG2026
Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
Shuchen Zhu, Rizhen Hu, Mingze Wang +4
Pre-training Large Language Models requires immense computational resources, making optimizer efficiency essential. The optimization landscape is highly anisotropic, with loss redu…