Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning

Prakhar Gupta, Garv Shah, Satyam Goyal +1

Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) t…