Approximate Muon with low-rank adapters

Ben Anson, Conor Houghton, Edward Milsom

The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One po…