The Role of Pre-training Data in Transfer Learning

Rahim Entezari, Mitchell Wortsman, Olga Saukh +3

The transfer learning paradigm of model pre-training and subsequent fine-tuning produces high-accuracy models. While most studies recommend scaling the pre-training size to benefit…