FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning

Yequan Zhao, Ruijie Zhang, Liyan Tan +3

Both full fine-tuning (Full FT) and parameter-efficient fine-tuning methods such as LoRA introduce weight updates without accounting for the spectral structure established during p…