Multilingual Neural Machine Translation with Knowledge Distillation

Abstract

Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation. Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method. Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.

Accepted by ICLR 2019

Cited by in corpus (53)

• Knowledge Distillation: A Survey
• Adaptive Multi-Teacher Multi-level Knowledge Distillation
• A Survey on Neural Speech Synthesis
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• Bridging the Modality Gap for Speech-to-Text Translation
• End-to-End Speech Translation with Knowledge Distillation
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
• Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
• A Comprehensive Survey of Multilingual Neural Machine Translation
• Teacher-Student Framework Enhanced Multi-domain Dialogue Generation
• What is Next when Sequential Prediction Meets Implicitly Hard Interaction?
• Distilling Knowledge Learned in BERT for Text Generation
• Imitation Attacks and Defenses for Black-box Machine Translation Systems
• Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
• Multilingual Neural Machine Translation with Language Clustering
• Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language
• Noisy Self-Knowledge Distillation for Text Summarization
• A Study of Multilingual Neural Machine Translation
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
• Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation
• Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
• Dealing with training and test segmentation mismatch: FBK@IWSLT2021
• Synchronous Bidirectional Learning for Multilingual Lip Reading
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Evaluating Explanation Methods for Neural Machine Translation
• A Brief Survey of Multilingual Neural Machine Translation
• Distilling the Knowledge of BERT for Sequence-to-Sequence ASR
• Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers
• Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains
• Structure-Level Knowledge Distillation For Multilingual Sequence Labeling
• Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation
• Go From the General to the Particular: Multi-Domain Translation with Domain Transformation Networks
• Learning from a Lightweight Teacher for Efficient Knowledge Distillation
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
• Selective Knowledge Distillation for Neural Machine Translation
• A Study of Non-autoregressive Model for Sequence Generation
• Dynamically Composing Domain-Data Selection with Clean-Data Selection by "Co-Curricular Learning" for Neural Machine Translation
• A Survey on Green Deep Learning
• Hierarchical Knowledge Distillation for Dialogue Sequence Labeling
• More Parameters? No Thanks!
• Multilingual Translation via Grafting Pre-trained Language Models
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
• Multilingual AMR Parsing with Noisy Knowledge Distillation
• Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition
• Multilingual Neural Machine Translation:Can Linguistic Hierarchies Help?
• Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling
• Ensemble Knowledge Distillation for CTR Prediction
• Iterative Dual Domain Adaptation for Neural Machine Translation
• Counter-Interference Adapter for Multilingual Machine Translation