Ensemble Distillation for Neural Machine Translation

Abstract

Knowledge distillation describes a method for training a student network to perform better by learning from a stronger teacher network. Translating a sentence with an Neural Machine Translation (NMT) engine is time expensive and having a smaller model speeds up this process. We demonstrate how to transfer the translation quality of an ensemble and an oracle BLEU teacher network into a single NMT system. Further, we present translation improvements from a teacher network that has the same architecture and dimensions of the student network. As the training of the student model is still expensive, we introduce a data filtering method based on the knowledge of the teacher model that not only speeds up the training, but also leads to better translation quality. Our techniques need no code change and can be easily reproduced with any NMT architecture to speed up the decoding process.

References in corpus (3)

• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• Sequence-Level Knowledge Distillation

Cited by in corpus (26)

• Knowledge Distillation: A Survey
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Data Diversification: A Simple Strategy For Neural Machine Translation
• Bridging the Modality Gap for Speech-to-Text Translation
• End-to-End Speech Translation with Knowledge Distillation
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• Cross-model Back-translated Distillation for Unsupervised Machine Translation
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
• Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition
• Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers
• Knowledge Distillation as Semiparametric Inference
• Selective Knowledge Distillation for Neural Machine Translation
• Neural Machine Translation: A Review and Survey
• Balancing Cost and Benefit with Tied-Multi Transformers
• The NiuTrans Machine Translation Systems for WMT21
• Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs
• Fully Synthetic Data Improves Neural Machine Translation with Knowledge Distillation
• Multilingual AMR Parsing with Noisy Knowledge Distillation
• The Volctrans Neural Speech Translation System for IWSLT 2021
• Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translation
• Hierarchical Knowledge Distillation for Dialogue Sequence Labeling
• Text Simplification by Tagging
• Reciprocal Supervised Learning Improves Neural Machine Translation
• Weight Distillation: Transferring the Knowledge in Neural Network Parameters
• Learning Metrics from Mean Teacher: A Supervised Learning Method for Improving the Generalization of Speaker Verification System
• Multi-Layer Softmaxing during Training Neural Machine Translation for Flexible Decoding with Fewer Layers