When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting

Vedant Palit, Florent Draye, Nicolas Zucchet +2

Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can…