Soup to go: mitigating forgetting during continual learning with model averaging

Anat Kleiman, Gintare Karolina Dziugaite, Jonathan Frankle +2

In continual learning, where task data arrives in a sequence, fine-tuning on later tasks will often lead to performance degradation on earlier tasks. This is especially pronounced…