Insights on Muon from Simple Quadratics

Antoine Gonon, Andreea-Alexandra Muşat, Nicolas Boumal

Muon updates weight matrices along (approximate) polar factors of the gradients and has shown strong empirical performance in large-scale training. Existing attempts at explaining…