SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity

Samir Khaki, Xiuyu Li, Junxian Guo +7

Fine-tuning LLMs is both computationally and memory-intensive. While parameter-efficient fine-tuning methods, such as QLoRA and DoRA, reduce the number of trainable parameters and…