Selective Knowledge Distillation for Neural Machine Translation

Abstract

Neural Machine Translation (NMT) models achieve state-of-the-art performance on many translation benchmarks. As an active research field in NMT, knowledge distillation is widely applied to enhance the model's performance by transferring teacher model's knowledge on each training sample. However, previous work rarely discusses the different impacts and connections among these samples, which serve as the medium for transferring teacher knowledge. In this paper, we design a novel protocol that can effectively analyze the different impacts of samples by comparing various samples' partitions. Based on above protocol, we conduct extensive experiments and find that the teacher's knowledge is not the more, the better. Knowledge over specific samples may even hurt the whole performance of knowledge distillation. Finally, to address these issues, we propose two simple yet effective strategies, i.e., batch-level and global-level selections, to pick suitable samples for distillation. We evaluate our approaches on two large-scale machine translation tasks, WMT'14 English->German and WMT'19 Chinese->English. Experimental results show that our approaches yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively.

Accepted as a long paper at ACL 2021. Code is available at https://github.com/LeslieOverfitting/selective_distillation

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• The Evolved Transformer
• Multilingual Neural Machine Translation with Knowledge Distillation
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation