The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning

Peng Xie, Amr Alanwar

LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix $W+BA$ that are nearly orthogonal to all pretrained singular vectors and…