On the Connection between Pre-training Data Diversity and Fine-tuning Robustness

Vivek Ramanujan, Thao Nguyen, Sewoong Oh +2

Pre-training has been widely adopted in deep learning to improve model performance, especially when the training data for a target task is limited. In our work, we seek to understa…