Toward a First-Principles Update Geometry for the Language-Model Head

Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2

Muon motivates designing optimizer geometry around the function of each parameter block and uses the spectral norm for hidden linear layers. For the language-model head, the spectr…