Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer +2

How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabili…