On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets

Cheng-Han Chiang, Hung-yi Lee

Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance than their counterparts directly tra…