GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters

Anand Choudhary, Yasser Sulaıman, Lukas Mauch +3

Sparse fine-tuning techniques adapt LLMs to downstream tasks by only tuning a sparse subset of model parameters. However, the effectiveness of sparse adaptation depends on optimall…