Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation

Abstract

Sequence-level knowledge distillation (SLKD) is a model compression technique that leverages large, accurate teacher models to train smaller, under-parameterized student models. Why does pre-processing MT data with SLKD help us train smaller models? We test the common hypothesis that SLKD addresses a capacity deficiency in students by "simplifying" noisy data points and find it unlikely in our case. Models trained on concatenations of original and "simplified" datasets generalize just as well as baseline SLKD. We then propose an alternative hypothesis under the lens of data augmentation and regularization. We try various augmentation strategies and observe that dropout regularization can become unnecessary. Our methods achieve BLEU gains of 0.7-1.2 on TED Talks.

References in corpus (3)

• Distilling the Knowledge in a Neural Network
• Sockeye: A Toolkit for Neural Machine Translation
• Mathematics of Deep Learning

Cited by in corpus (6)

• Knowledge Distillation: A Survey
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation
• Selective Knowledge Distillation for Neural Machine Translation
• ESPnet-ST IWSLT 2021 Offline Speech Translation System
• Weight Distillation: Transferring the Knowledge in Neural Network Parameters