Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges

Abstract

We introduce our efforts towards building a universal neural machine translation (NMT) system capable of translating between any language pair. We set a milestone towards this goal by building a single massively multilingual NMT model handling 103 languages trained on over 25 billion examples. Our system demonstrates effective transfer learning ability, significantly improving translation quality of low-resource languages, while keeping high-resource language translation quality on-par with competitive bilingual baselines. We provide in-depth analysis of various aspects of model building that are crucial to achieving quality and practicality in universal NMT. While we prototype a high-quality universal translation system, our extensive empirical analysis exposes issues that need to be further addressed, and we suggest directions for future research.

References in corpus (49)

• Sequence to Sequence Learning with Neural Networks
• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Methods for Interpreting and Understanding Deep Neural Networks
• Convolutional Sequence to Sequence Learning
• An Overview of Multi-Task Learning in Deep Neural Networks
• Theoretical Models of Learning to Learn
• A Survey on Multi-Task Learning
• Achieving Human Parity on Automatic Chinese to English News Translation
• On First-Order Meta-Learning Algorithms
• On Using Monolingual Corpora in Neural Machine Translation
• Non-Autoregressive Neural Machine Translation
• Deep Learning Scaling is Predictable, Empirically
• Massively Multitask Networks for Drug Discovery
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Training deep neural networks with low precision multiplications
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• One Model To Learn Them All
• Character-based Neural Machine Translation
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Parameter-Efficient Transfer Learning for NLP
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Fixup Initialization: Residual Learning Without Normalization
• Understanding Back-Translation at Scale
• The Benefit of Multitask Representation Learning
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Unsupervised Neural Machine Translation
• The Missing Ingredient in Zero-Shot Neural Machine Translation
• Continual Learning via Neural Pruning
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• SYSTRAN's Pure Neural Machine Translation Systems
• Online Learning Rate Adaptation with Hypergradient Descent
• Multilingual Neural Machine Translation with Task-Specific Attention
• Theory and Experiments on Vector Quantized Autoencoders
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Adaptive Scheduling for Multi-Task Learning
• Meta-Learning for Low-Resource Neural Machine Translation
• Learning to Segment Inputs for NMT Favors Character-Level Processing
• Multilingual Language Processing From Bytes
• A neural interlingua for multilingual machine translation
• Training Deeper Neural Machine Translation Models with Transparent Attention
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection
• Contextual Parameter Generation for Universal Neural Machine Translation
• Towards Interlingua Neural Machine Translation

Cited by in corpus (69)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Multilingual Denoising Pre-training for Neural Machine Translation
• mT5: A massively multilingual pre-trained text-to-text transformer
• Beyond English-Centric Multilingual Machine Translation
• A Survey on Negative Transfer
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• IndicBART: A Pre-trained Model for Indic Natural Language Generation
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Finetuned Language Models Are Zero-Shot Learners
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
• CoVoST 2 and Massively Multilingual Speech-to-Text Translation
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• A Comprehensive Survey of Multilingual Neural Machine Translation
• Deep Transformers with Latent Depth
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Low Resource Neural Machine Translation: A Benchmark for Five African Languages
• Scaling Laws for Neural Machine Translation
• Nearest Neighbor Machine Translation
• SQuId: Measuring Speech Naturalness in Many Languages
• Scaling End-to-End Models for Large-Scale Multilingual ASR
• Adaptive Sparse Transformer for Multilingual Translation
• Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts
• Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
• Marathi To English Neural Machine Translation With Near Perfect Corpus And Transformers
• Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• The Effect of Domain and Diacritics in Yorùbá-English Neural Machine Translation
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Can Monolingual Pretrained Models Help Cross-Lingual Classification?
• Including Signed Languages in Natural Language Processing
• An Open Dataset and Model for Language Identification
• Lite Training Strategies for Portuguese-English and English-Portuguese Translation
• Stochastic Optimization with Laggard Data Pipelines
• The 2020s Political Economy of Machine Translation
• Multi-task Learning for Multilingual Neural Machine Translation
• A Survey on Low-Resource Neural Machine Translation
• Neural machine translation, corpus and frugality
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
• Cross-lingual Inductive Transfer to Detect Offensive Language
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• Continual Learning in Multilingual NMT via Language-Specific Embeddings
• Sicilian Translator: A Recipe for Low-Resource NMT
• Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning
• Multilingual Medical Question Answering and Information Retrieval for Rural Health Intelligence Access
• MergeDistill: Merging Pre-trained Language Models using Distillation
• Revisiting Modularized Multilingual NMT to Meet Industrial Demands
• ESPnet-ST IWSLT 2021 Offline Speech Translation System
• Multilingual Translation via Grafting Pre-trained Language Models
• Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits
• Multilingual AMR-to-Text Generation
• Self-Learning for Zero Shot Neural Machine Translation
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
• Improving Multilingual Translation by Representation and Gradient Regularization
• One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks
• XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
• More Parameters? No Thanks!
• Distributionally Robust Multilingual Machine Translation
• Competence-based Curriculum Learning for Multilingual Machine Translation
• AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages