Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success

Florian Hübler, Thomas Pethick, Suvrit Sra

Non-Euclidean optimisation methods with matrix-valued updates, such as Muon and Scion, have recently shown strong empirical performance for training Transformer models, yet their t…