Muon${}^2$: Boosting Muon via Adaptive Second-Moment Preconditioning

Ziyue Liu, Ruijie Zhang, Zhengyang Wang +4

Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonaliza…