Are Large-scale Datasets Necessary for Self-Supervised Pre-training?

Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron +3

Pre-training models on large scale datasets, like ImageNet, is a standard practice in computer vision. This paradigm is especially effective for tasks with small training sets, for…