Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection

Chenglong Wang, Yi Lu, Yongyu Mu +3

Knowledge distillation addresses the problem of transferring knowledge from a teacher model to a student model. In this process, we typically have multiple types of knowledge extra…