Pre-training via Paraphrasing

Abstract

We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual multi-document paraphrasing objective. MARGE provides an alternative to the dominant masked language modeling paradigm, where we self-supervise the reconstruction of target text by retrieving a set of related texts (in many languages) and conditioning on them to maximize the likelihood of generating the original. We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization. The objective noisily captures aspects of paraphrase, translation, multi-document summarization, and information retrieval, allowing for strong zero-shot performance on several tasks. For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation. We further show that fine-tuning gives strong performance on a range of discriminative and generative tasks in many languages, making MARGE the most generally applicable pre-training method to date.

References in corpus (16)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Cross-lingual Language Model Pretraining
• Scaling Laws for Neural Language Models
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Multilingual Denoising Pre-training for Neural Machine Translation
• REALM: Retrieval-Augmented Language Model Pre-Training
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Unsupervised Neural Machine Translation
• No Training Required: Exploring Random Encoders for Sentence Classification
• Generating Wikipedia by Summarizing Long Sequences
• Generalization through Memorization: Nearest Neighbor Language Models
• Controllable Abstractive Summarization

Cited by in corpus (28)

• mT5: A massively multilingual pre-trained text-to-text transformer
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Pretrained Transformers for Text Ranking: BERT and Beyond
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Distilling Knowledge from Reader to Retriever for Question Answering
• Support-set bottlenecks for video-text representation learning
• WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
• Neural Language Generation: Formulation, Methods, and Evaluation
• Better Fine-Tuning by Reducing Representational Collapse
• GNN-LM: Language Modeling based on Global Contexts via GNN
• MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
• Neural Machine Translation with Monolingual Translation Memory
• Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning
• MS2: Multi-Document Summarization of Medical Studies
• LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training
• Mention Memory: incorporating textual knowledge into Transformers through entity mention attention
• Current Limitations of Language Models: What You Need is Retrieval
• Robust Transfer Learning with Pretrained Language Models through Adapters
• Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning
• Summarizing Utterances from Japanese Assembly Minutes using Political Sentence-BERT-based Method for QA Lab-PoliInfo-2 Task of NTCIR-15
• Cross-Thought for Sentence Encoder Pre-training
• Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning
• Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
• Data-to-text Generation by Splicing Together Nearest Neighbors
• VisualSem: A High-quality Knowledge Graph for Vision and Language
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals