LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold

Franz Louis Cesista, Katherine Crowson, Cédric Simal +1

Low-Rank Adaptation (LoRA) significantly reduces compute and memory costs for finetuning Deep Learning models but is often harder to tune than dense training: when using factor-wis…