One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

Chuhan Wu, Fangzhao Wu, Yongfeng Huang

Pre-trained language models (PLMs) achieve great success in NLP. However, their huge model sizes hinder their applications in many practical systems. Knowledge distillation is a po…