Replaying pre-training data improves fine-tuning

Suhas Kotha, Percy Liang

To obtain a language model for a target domain (e.g. math), the current paradigm is to pre-train on a vast amount of generic web text and then fine-tune on the relatively limited a…