Muon is Scalable for LLM Training

Jingyuan Liu, Jianlin Su, Xingcheng Yao +25

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not…