MASS: Masked Sequence to Sequence Pre-training for Language Generation

Abstract

Pre-training and fine-tuning, e.g., BERT, have achieved great success in language understanding by transferring knowledge from rich-resource pre-training task to the low/zero-resource downstream tasks. Inspired by the success of BERT, we propose MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder based language generation tasks. MASS adopts the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence: its encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and its decoder tries to predict this masked fragment. In this way, MASS can jointly train the encoder and decoder to develop the capability of representation extraction and language modeling. By further fine-tuning on a variety of zero/low-resource language generation tasks, including neural machine translation, text summarization and conversational response generation (3 tasks and totally 8 datasets), MASS achieves significant improvements over the baselines without pre-training or with other pre-training methods. Specially, we achieve the state-of-the-art accuracy (37.5 in terms of BLEU score) on the unsupervised English-French translation, even beating the early attention-based supervised model.

Accepted by ICML 2019

Cited by in corpus (219)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Pre-trained Models for Natural Language Processing: A Survey
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• MPNet: Masked and Permuted Pre-training for Language Understanding
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
• A Survey of Knowledge-Enhanced Text Generation
• CLEAR: Contrastive Learning for Sentence Representation
• Unifying Vision-and-Language Tasks via Text Generation
• Incorporating BERT into Neural Machine Translation
• Span-based Joint Entity and Relation Extraction with Transformer Pre-training
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Revisiting Self-Training for Neural Sequence Generation
• Paradigm Shift in Natural Language Processing
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• Low-Resource Knowledge-Grounded Dialogue Generation
• A Mutual Information Maximization Perspective of Language Representation Learning
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Generating Persona Consistent Dialogues by Exploiting Natural Language Inference
• Unsupervised Translation of Programming Languages
• Text Compression-aided Transformer Encoding
• A Call for More Rigor in Unsupervised Cross-lingual Learning
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
• M6: A Chinese Multimodal Pretrainer
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• I-MAD: Interpretable Malware Detector Using Galaxy Transformer
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Learning to summarize from human feedback
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures
• Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• When Does Unsupervised Machine Translation Work?
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
• Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models
• FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition
• ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
• A Survey of Knowledge Enhanced Pre-trained Models
• XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
• Neural Language Generation: Formulation, Methods, and Evaluation
• A Targeted Attack on Black-Box Neural Machine Translation with Parallel Data Poisoning
• Encoder-Agnostic Adaptation for Conditional Language Generation
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning
• PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
• TreeBERT: A Tree-Based Pre-Trained Model for Programming Language
• Distilling Knowledge Learned in BERT for Text Generation
• Improving Readability for Automatic Speech Recognition Transcription
• Coreferential Reasoning Learning for Language Representation
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
• GNN-LM: Language Modeling based on Global Contexts via GNN
• DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
• SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
• Weakly-Supervised Video Moment Retrieval via Semantic Completion Network
• Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarization
• A Survey on Document-level Neural Machine Translation: Methods and Evaluation
• Zero-Shot Paraphrase Generation with Multilingual Language Models
• Can Unconditional Language Models Recover Arbitrary Sentences?
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
• Pretrained Language Models for Text Generation: A Survey
• Multimodal Pretraining for Dense Video Captioning
• Mathematical Reasoning via Self-supervised Skip-tree Training
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• The Life Cycle of Knowledge in Big Language Models: A Survey
• SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization
• Multilingual Neural Machine Translation with Language Clustering
• Cross-model Back-translated Distillation for Unsupervised Machine Translation
• Noisy Self-Knowledge Distillation for Text Summarization
• Acquiring Knowledge from Pre-trained Model to Neural Machine Translation
• s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
• A Unified Generative Framework for Aspect-Based Sentiment Analysis
• Emergent Communication Pretraining for Few-Shot Machine Translation
• On Learning Universal Representations Across Languages
• Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
• Cross-Lingual Natural Language Generation via Pre-Training
• MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training
• Graph-based Multi-hop Reasoning for Long Text Generation
• When and Why is Unsupervised Neural Machine Translation Useless?
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
• On the Effectiveness of Pretrained Models for API Learning
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning
• GLGE: A New General Language Generation Evaluation Benchmark
• Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
• Language Graph Distillation for Low-Resource Machine Translation
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
• Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
• A Study of Multilingual Neural Machine Translation
• Diverse Pretrained Context Encodings Improve Document Translation
• Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
• Generating Representative Headlines for News Stories
• Mask Attention Networks: Rethinking and Strengthen Transformer
• QURIOUS: Question Generation Pretraining for Text Generation
• Conditional Text Generation for Harmonious Human-Machine Interaction
• PopMAG: Pop Music Accompaniment Generation
• ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation
• Independent language modeling architecture for end-to-end ASR
• Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification
• Future Data Helps Training: Modeling Future Contexts for Session-based Recommendation
• Generating Code with the Help of Retrieved Template Functions and Stack Overflow Answers
• Proof Artifact Co-training for Theorem Proving with Language Models
• Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
• Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification using Pre-trained Language Models
• Denoising based Sequence-to-Sequence Pre-training for Text Generation
• Measuring Patent Claim Generation by Span Relevancy
• A Unified Generative Framework for Various NER Subtasks
• Which *BERT? A Survey Organizing Contextualized Encoders
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Reference Language based Unsupervised Neural Machine Translation
• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
• Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events
• An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models
• Rethinking Perturbations in Encoder-Decoders for Fast Training
• Logic2Text: High-Fidelity Natural Language Generation from Logical Forms
• Conditional Augmentation for Aspect Term Extraction via Masked Sequence-to-Sequence Generation
• Pre-training for Abstractive Document Summarization by Reinstating Source Text
• Code-switching pre-training for neural machine translation
• Unsupervised Parallel Corpus Mining on Web Data
• Unsupervised Translation of German--Lower Sorbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language
• Enhancing Semantic Understanding with Self-supervised Methods for Abstractive Dialogue Summarization
• Plug and Play Autoencoders for Conditional Text Generation
• Attention Head Masking for Inference Time Content Selection in Abstractive Summarization
• Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios
• AutoTrans: Automating Transformer Design via Reinforced Architecture Search
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
• DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
• Discovering Useful Sentence Representations from Large Pretrained Language Models
• Softmax Tempering for Training Neural Machine Translation Models
• Narrative Incoherence Detection
• Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Explicit Sentence Compression for Neural Machine Translation
• Unsupervised Pivot Translation for Distant Languages
• Cross-lingual Supervision Improves Unsupervised Neural Machine Translation
• UWSpeech: Speech to Speech Translation for Unwritten Languages
• Unsupervised Text Summarization via Mixed Model Back-Translation
• Pre-training via Leveraging Assisting Languages and Data Selection for Neural Machine Translation
• Efficient Bidirectional Neural Machine Translation
• A Survey on Low-Resource Neural Machine Translation
• Paraphrase Generation as Unsupervised Machine Translation
• Cross-Lingual Abstractive Summarization with Limited Parallel Resources
• Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data
• LenAtten: An Effective Length Controlling Unit For Text Summarization
• Advances in Multi-turn Dialogue Comprehension: A Survey
• Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
• Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots
• Meta Back-translation
• Rapid Domain Adaptation for Machine Translation with Monolingual Data
• ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
• Few-Shot Learning of an Interleaved Text Summarization Model by Pretraining with Synthetic Data
• JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation
• Towards Machine Translation for the Kurdish Language
• MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
• The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task
• Explicit Cross-lingual Pre-training for Unsupervised Machine Translation
• nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
• Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
• Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study
• On the Copying Behaviors of Pre-Training for Neural Machine Translation
• K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce
• Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries
• Automatic Standardization of Colloquial Persian
• MAPGN: MAsked Pointer-Generator Network for sequence-to-sequence pre-training
• Investigating Label Bias in Beam Search for Open-ended Text Generation
• Interest-Behaviour Multiplicative Network for Resource-limited Recommendation
• A New Approach to Overgenerating and Scoring Abstractive Summaries
• A Self-supervised Approach for Semantic Indexing in the Context of COVID-19 Pandemic
• Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
• Enriching and Controlling Global Semantics for Text Summarization
• Explaining Documents' Relevance to Search Queries
• Empirical Analysis of Korean Public AI Hub Parallel Corpora and in-depth Analysis using LIWC
• On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation
• Examining the rhetorical capacities of neural language models
• CITIES: Contextual Inference of Tail-Item Embeddings for Sequential Recommendation
• Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance
• Exploiting Curriculum Learning in Unsupervised Neural Machine Translation
• Personalized Patent Claim Generation and Measurement
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation
• Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization
• Multilingual Translation via Grafting Pre-trained Language Models
• Sentence Bottleneck Autoencoders from Transformer Language Models
• AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages
• HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints
• Unsupervised Pre-training with Structured Knowledge for Improving Natural Language Inference
• Frustratingly Simple Pretraining Alternatives to Masked Language Modeling
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey
• Inference Time Style Control for Summarization
• Intent Classification and Slot Filling for Privacy Policies
• Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
• Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation
• Improving Truthfulness of Headline Generation
• "Wikily" Supervised Neural Translation Tailored to Cross-Lingual Tasks
• Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables
• A Generative Approach to Titling and Clustering Wikipedia Sections
• Reciprocal Supervised Learning Improves Neural Machine Translation
• Do all Roads Lead to Rome? Understanding the Role of Initialization in Iterative Back-Translation
• Robust Unsupervised Neural Machine Translation with Adversarial Denoising Training
• Multilingual AMR-to-Text Generation
• Neural Machine Translation with Error Correction
• Can You Traducir This? Machine Translation for Code-Switched Input
• Dialogue Inspectional Summarization with Factual Inconsistency Awareness
• End to End Dialogue Transformer
• Scrambled Translation Problem: A Problem of Denoising UNMT
• Scheduled Sampling Based on Decoding Steps for Neural Machine Translation