These Are Not All the Features You Are Looking For: A Fundamental Bottleneck in Supervised Pretraining

Xingyu Alice Yang, Jianyu Zhang, Léon Bottou

Transfer learning is widely used to adapt large pretrained models to new tasks with only a small amount of new data. However, a challenge persists -- the features from the original…