Investigating Transferability in Pretrained Language Models

Alex Tamkin, Trisha Singh, Davide Giovanardi +1

How does language model pretraining help transfer learning? We consider a simple ablation technique for determining the impact of each pretrained layer on transfer task performance…