Unraveling Key Factors of Knowledge Distillation

Abstract

Knowledge distillation, a technique for model compression and performance enhancement, has gained significant traction in Neural Machine Translation (NMT). However, existing research primarily focuses on empirical applications, and there is a lack of comprehensive understanding of how student model capacity, data complexity, and decoding strategies collectively influence distillation effectiveness. Addressing this gap, our study conducts an in-depth investigation into these factors, particularly focusing on their interplay in word-level and sequence-level distillation within NMT. Through extensive experimentation across datasets like IWSLT13 En$\to$Fr, IWSLT14 En$\to$De, and others, we empirically validate hypotheses related to the impact of these factors on knowledge distillation. Our research not only elucidates the significant influence of model capacity, data complexity, and decoding strategies on distillation effectiveness but also introduces a novel, optimized distillation approach. This approach, when applied to the IWSLT14 de$\to$en translation task, achieves state-of-the-art performance, demonstrating its practical efficacy in advancing the field of NMT.

I am requesting the withdrawal of this paper from arXiv due to the realization that the overall composition and structure of the article are not yet sufficiently refined. It is my intention to thoroughly revise and enhance the paper to ensure that it meets the highest standards of academic writing and accurately reflects the research conducted