Beyond the Matrix Sign: Quadratic Spectral Descent

Qiaozhe Zhang, Jun Sun, Yingzhuang Liu

Muon emerges as a strong competitor of the AdamW for LLM pretraining, because the matrix-wise update it employs can potentially incur smaller second-order penalty than the once dom…