3 papers
cs.LG2026
The Role of Quantization in Loss Spikes
Zaiqian Chen, Aditya Somasundaram, Sophie Li +4
Loss spikes in language model pretraining can waste substantial compute and derail otherwise healthy training runs. A primary driver of these instabilities is the growth and diverg…
cs.LG2026
Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry
Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2
Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearit…
cs.LG2026
Toward a First-Principles Update Geometry for the Language-Model Head
Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2
Muon motivates designing optimizer geometry around the function of each parameter block and uses the spectral norm for hidden linear layers. For the language-model head, the spectr…