A Note on the Convergence of Muon

Jiaxiang Li, Mingyi Hong

In this note, we inspect the convergence of a new optimizer for pretraining LLMs, namely the Muon optimizer. Such an optimizer is closely related to a specialized steepest descent…