Scaling Sparse Fine-Tuning to Large Language Models

Alan Ansell, Ivan Vulić, Hannah Sterz +2

Large Language Models (LLMs) are difficult to fully fine-tune (e.g., with instructions or human feedback) due to their sheer number of parameters. A family of parameter-efficient s…