Aurora: A Leverage-Aware Spectral Optimizer

Alec Dewulf, Dhruv Pai, Li Yang +2

We show that for tall matrix parameters, like projection matrices in the MLP layers, the Muon update can have row norms that are arbitrarily non-uniform. This can lead to a self-re…