Beyond English-Centric Multilingual Machine Translation

Abstract

Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages. However, much of this work is English-Centric by training only on data which was translated from or to English. While this is supported by large sources of training data, it does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT. We open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.

References in corpus (19)

• Convolutional Sequence to Sequence Learning
• Scaling Laws for Neural Language Models
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Multilingual Denoising Pre-training for Neural Machine Translation
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Reducing Transformer Depth on Demand with Structured Dropout
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
• Towards Supervised and Unsupervised Neural Machine Translation Baselines for Nigerian Pidgin
• Igbo-English Machine Translation: An Evaluation Benchmark
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

Cited by in corpus (45)

• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
• KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
• Scalable and Efficient MoE Training for Multitask Multilingual Models
• GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation
• Towards Better Chinese-centric Neural Machine Translation for Low-resource Languages
• RuCoLA: Russian Corpus of Linguistic Acceptability
• Factual Consistency of Multilingual Pretrained Language Models
• Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models
• Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
• HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation
• Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
• Investigating Failures of Automatic Translation in the Case of Unambiguous Gender
• Improving Zero-Shot Translation by Disentangling Positional Information
• Translation-Enhanced Multilingual Text-to-Image Generation
• Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset
• DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages
• The Effect of Domain and Diacritics in Yorùbá-English Neural Machine Translation
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Sometimes We Want Translationese
• Learning Language Specific Sub-network for Multilingual Machine Translation
• Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
• Transfer Learning for Multi-lingual Tasks -- a Survey
• Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
• Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data
• Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
• EdinSaar@WMT21: North-Germanic Low-Resource Multilingual NMT
• Sicilian Translator: A Recipe for Low-Resource NMT
• The JHU-Microsoft Submission for WMT21 Quality Estimation Shared Task
• DeepRepair: Style-Guided Repairing for DNNs in the Real-world Operational Environment
• Hierarchical Transformer for Multilingual Machine Translation
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space
• CUNI systems for WMT21: Multilingual Low-Resource Translation for Indo-European Languages Shared Task
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
• A Survey on Green Deep Learning
• An Empirical Investigation of Multi-bridge Multilingual NMT models
• Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer
• Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
• FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task
• Attentive fine-tuning of Transformers for Translation of low-resourced languages @LoResMT 2021
• Discovering Representation Sprachbund For Multilingual Pre-Training
• Competence-based Curriculum Learning for Multilingual Machine Translation
• Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation