Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation

Jingxuan Wei, Linzhuang Sun, Yichong Leng +3

Knowledge distillation, transferring knowledge from a teacher model to a student model, has emerged as a powerful technique in neural machine translation for compressing models or…