Pretraining Curricula Enable Selective Fine-tuning

Sebastian A. Bruijns, Jirko Rubruck, Mia H. Whitefield +3

Transformers follow implicit curricula whereby some tasks are learned before others. However, how explicit pretraining curricula influence learning, generalization, and the selecti…