Multilingual Denoising Pre-training for Neural Machine Translation

Abstract

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -- a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

Work in progress

References in corpus (27)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Cross-lingual Language Model Pretraining
• Exploiting Similarities among Languages for Machine Translation
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• On the Cross-lingual Transferability of Monolingual Representations
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Word Translation Without Parallel Data
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Phrase-Based & Neural Unsupervised Machine Translation
• Unsupervised Machine Translation Using Monolingual Corpora Only
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Text Summarization with Pretrained Encoders
• XNLI: Evaluating Cross-lingual Sentence Representations
• How multilingual is Multilingual BERT?
• Does Neural Machine Translation Benefit from Larger Context?
• Unsupervised Neural Machine Translation
• Universal Neural Machine Translation for Extremely Low Resource Languages
• Massively Multilingual Neural Machine Translation
• Exploiting Cross-Sentence Context for Neural Machine Translation
• Towards Making the Most of BERT in Neural Machine Translation
• Pre-trained Language Model Representations for Language Generation
• Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models
• Pretrained Language Models for Document-Level Neural Machine Translation
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Translating Translationese: A Two-Step Approach to Unsupervised Machine Translation

Cited by in corpus (136)

• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• Pre-trained Models for Natural Language Processing: A Survey
• Beyond English-Centric Multilingual Machine Translation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Towards a Unified View of Parameter-Efficient Transfer Learning
• Natural Language Processing Advancements By Deep Learning: A Survey
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• Pre-training via Paraphrasing
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Finetuned Language Models Are Zero-Shot Learners
• Pre-trained Summarization Distillation
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
• A Call for More Rigor in Unsupervised Cross-lingual Learning
• LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• Hierarchical Learning for Generation with Long Source Sequences
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling
• Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
• Scalable and Efficient MoE Training for Multitask Multilingual Models
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
• A Targeted Attack on Black-Box Neural Machine Translation with Parallel Data Poisoning
• Detecting Hallucinated Content in Conditional Neural Sequence Generation
• MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
• A Survey on Document-level Neural Machine Translation: Methods and Evaluation
• Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
• HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation
• Multi-Stage Pre-training for Low-Resource Domain Adaptation
• Cross-model Back-translated Distillation for Unsupervised Machine Translation
• Adaptive Sparse Transformer for Multilingual Translation
• On Learning Universal Representations Across Languages
• Code-switched inspired losses for generic spoken dialog representations
• Emergent Communication Pretraining for Few-Shot Machine Translation
• Empirical Analysis of Training Strategies of Transformer-based Japanese Chit-chat Systems
• When and Why is Unsupervised Neural Machine Translation Useless?
• Unsupervised Neural Machine Translation with Generative Language Models Only
• Direct Text to Speech Translation System using Acoustic Units
• Building a Parallel Corpus and Training Translation Models Between Luganda and English
• Mixed-Lingual Pre-training for Cross-lingual Summarization
• ITALIC: An Italian Intent Classification Dataset
• DIALOG-22 RuATD Generated Text Detection
• Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training
• One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval
• Automatic Summarization of Russian Texts: Comparison of Extractive and Abstractive Methods
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Strategies for improving low resource speech to text translation relying on pre-trained ASR models
• Are the Multilingual Models Better? Improving Czech Sentiment with Transformers
• ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
• YANMTT: Yet Another Neural Machine Translation Toolkit
• Autocorrect in the Process of Translation -- Multi-task Learning Improves Dialogue Machine Translation
• Which *BERT? A Survey Organizing Contextualized Encoders
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
• VieSum: How Robust Are Transformer-based Models on Vietnamese Summarization?
• Learning Language Specific Sub-network for Multilingual Machine Translation
• Unsupervised Translation of German--Lower Sorbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language
• Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
• Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
• SportsSum2.0: Generating High-Quality Sports News from Live Text Commentary
• Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings
• Revisiting Language Encoding in Learning Multilingual Representations
• Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation
• A Survey on Low-Resource Neural Machine Translation
• Machine Translation of Low-Resource Indo-European Languages
• Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation
• Cross-lingual Supervision Improves Unsupervised Neural Machine Translation
• Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning
• Rapid Domain Adaptation for Machine Translation with Monolingual Data
• ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
• Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data
• Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages
• Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling
• Cross-Lingual Training with Dense Retrieval for Document Retrieval
• Language Tags Matter for Zero-Shot Neural Machine Translation
• Paraphrase Generation as Unsupervised Machine Translation
• Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction
• Improving Neural Machine Translation by Bidirectional Training
• Generate & Rank: A Multi-task Framework for Math Word Problems
• Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
• Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision
• Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning
• Towards Machine Translation for the Kurdish Language
• nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
• Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study
• Positional Artefacts Propagate Through Masked Language Model Embeddings
• MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
• Cross-Lingual Language Model Meta-Pretraining
• Generalising Multilingual Concept-to-Text NLG with Language Agnostic Delexicalisation
• The USYD-JD Speech Translation System for IWSLT 2021
• Towards Universality in Multilingual Text Rewriting
• Unsupervised Machine Translation On Dravidian Languages
• On the Copying Behaviors of Pre-Training for Neural Machine Translation
• Don't Change Me! User-Controllable Selective Paraphrase Generation
• Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs
• Utilizing Resource-Rich Language Datasets for End-to-End Scene Text Recognition in Resource-Poor Languages
• Global-aware Beam Search for Neural Abstractive Summarization
• Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding
• An Unsupervised Method for Building Sentence Simplification Corpora in Multiple Languages
• FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task
• Deep Adversarially-Enhanced k-Nearest Neighbors
• From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text
• Making Better Use of Bilingual Information for Cross-Lingual AMR Parsing
• Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference
• Don't Go Far Off: An Empirical Study on Neural Poetry Translation
• A Dataset and Baselines for Multilingual Reply Suggestion
• AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages
• Investigating Code-Mixed Modern Standard Arabic-Egyptian to English Machine Translation
• Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages
• Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables
• Scrambled Translation Problem: A Problem of Denoising UNMT
• A Survey on Green Deep Learning
• Pretrained Language Models are Symbolic Mathematics Solvers too!
• Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation
• Multilingual Translation via Grafting Pre-trained Language Models
• Two-Headed Monster And Crossed Co-Attention Networks
• DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation
• Exploiting Curriculum Learning in Unsupervised Neural Machine Translation
• IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
• "Wikily" Supervised Neural Translation Tailored to Cross-Lingual Tasks
• Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation
• On Isotropy Calibration of Transformers
• On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation
• STIL -- Simultaneous Slot Filling, Translation, Intent Classification, and Language Identification: Initial Results using mBART on MultiATIS++
• Multilingual AMR Parsing with Noisy Knowledge Distillation
• PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation
• Empirical Analysis of Korean Public AI Hub Parallel Corpora and in-depth Analysis using LIWC
• RuSentEval: Linguistic Source, Encoder Force!
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
• Zero-Shot Language Transfer vs Iterative Back Translation for Unsupervised Machine Translation
• Do all Roads Lead to Rome? Understanding the Role of Initialization in Iterative Back-Translation