2 citations · 2 across the 3 of their papers we have counts for
3 papers · 1 filter
Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success
Florian Hübler, Thomas Pethick, Suvrit Sra
Non-Euclidean optimisation methods with matrix-valued updates, such as Muon and Scion, have recently shown strong empirical performance for training Transformer models, yet their t…
Can SGD Handle Heavy-Tailed Noise?
Ilyas Fatkhullin, Florian Hübler, Guanghui Lan
Stochastic Gradient Descent (SGD) is a cornerstone of large-scale optimization, yet its theoretical behavior under heavy-tailed noise -- common in modern machine learning and reinf…
From Gradient Clipping to Normalization for Heavy Tailed SGD
Florian Hübler, Ilyas Fatkhullin, Niao He
Recent empirical evidence indicates that many machine learning applications involve heavy-tailed gradient noise, which challenges the standard assumptions of bounded variance in st…