Tight Complexity of Muon under Heavy-Tailed Noise

Jiayu Zhang, Tianyi Lin

A growing lesson from neural network optimization is that optimizer design should respect how the model is parametrized. The layerwise input-output structure of neural networks mot…