Distilling Knowledge from Pre-trained Language Models via Text Smoothing

Xing Wu, Yibing Liu, Xiangyang Zhou +1

This paper studies compressing pre-trained language models, like BERT (Devlin et al.,2019), via teacher-student knowledge distillation. Previous works usually force the student mod…