Sparse Matrix in Large Language Model Fine-tuning

Haoze He, Juncheng Billy Li, Xuan Jiang +1

LoRA and its variants have become popular parameter-efficient fine-tuning (PEFT) methods due to their ability to avoid excessive computational costs. However, an accuracy gap often…