3 papers
cs.LG2026
MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration
Lianhai Ren, Yucheng Ding, Xiao Liu +3
Training instability remains a critical challenge in large language model (LLM) pretraining, often manifesting as sudden gradient explosions that waste significant computational re…
cs.CL2025
Sigma-MoE-Tiny Technical Report
Qingguo Hu, Zhenghao Lin, Ziyue Yang +12
Mixture-of-Experts (MoE) has emerged as a promising paradigm for foundation models due to its efficient and powerful scalability. In this work, we present Sigma-MoE-Tiny, an MoE la…
cs.DC2025
SIGMA: An AI-Empowered Training Stack on Early-Life Hardware
Lei Qu, Lianhai Ren, Peng Cheng +12
An increasing variety of AI accelerators is being considered for large-scale training. However, enabling large-scale training on early-life AI accelerators faces three core challen…