5 papers
MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads
Linkai Ma, Xinyu Luo, Mengbo Wang +3
The Muon optimizer derives its update rule for hidden linear layers by solving a local linearization of the loss penalized by the spectral norm, motivated by an RMS-stability argum…
The Advantages of Fresh Sketching for Ridge Regression
Linkai Ma, Qilin Li, Petros Drineas
Over the past 25 years, sketching and sampling have become widely used tools for accelerating large-scale regression. In iterative randomized solvers, a basic design choice is whet…
Stochastic Rounding Increases Small Singular Values
Linkai Ma, Tingzhou Yu, Petros Drineas
Over the past half-dozen years, stochastic rounding (SR) has regained significant attention as a quantization scheme for low-precision floating-point arithmetic, with applications…
A Note on the Stability of the Sherman-Morrison-Woodbury Formula
Linkai Ma, Christos Boutsikas, Mehrdad Ghadiri +1
We study the numerical stability of the Sherman-Morrison-Woodbury (SMW) identity. Let and assume and both have full-column rank. We explore error bounds for…
Stochastic Rounding Implicitly Regularizes Tall-and-Thin Matrices
Gregory Dexter, Christos Boutsikas, Linkai Ma +2
Motivated by the popularity of stochastic rounding in the context of machine learning and the training of large-scale deep neural network models, we consider stochastic nearness ro…