UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

Abstract

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM). Given an input text with masked tokens, we rely on conventional masks to learn inter-relations between corrupted tokens and context via autoencoding, and pseudo masks to learn intra-relations between masked spans via partially autoregressive modeling. With well-designed position embeddings and self-attention masks, the context encodings are reused to avoid redundant computation. Moreover, conventional masks used for autoencoding provide global masking information, so that all the position embeddings are accessible in partially autoregressive language modeling. In addition, the two tasks pre-train a unified language model as a bidirectional encoder and a sequence-to-sequence decoder, respectively. Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.

11 pages

References in corpus (5)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• A Simple Method for Commonsense Reasoning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Cited by in corpus (48)

• Pre-trained Models for Natural Language Processing: A Survey
• BEiT: BERT Pre-Training of Image Transformers
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• Deep Learning Based Text Classification: A Comprehensive Review
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• TransPolymer: a Transformer-based language model for polymer property predictions
• Swin Transformer V2: Scaling Up Capacity and Resolution
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Adversarial Training for Large Neural Language Models
• TAPEX: Table Pre-training via Learning a Neural SQL Executor
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• Rethinking Positional Encoding in Language Pre-training
• LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph
• NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction
• Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
• A Survey of Knowledge Enhanced Pre-trained Models
• KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning
• PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
• A Condense-then-Select Strategy for Text Summarization
• NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints
• Noisy Self-Knowledge Distillation for Text Summarization
• Unified Question Generation with Continual Lifelong Learning
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Accelerating Real-Time Question Answering via Question Generation
• To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
• An Enhanced Knowledge Injection Model for Commonsense Generation
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
• Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning
• Generating Query Focused Summaries from Query-Free Resources
• An Approach to Improve Robustness of NLP Systems against ASR Errors
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
• TAG: Gradient Attack on Transformer-based Language Models
• Open-Domain Dialogue Generation Based on Pre-trained Language Models
• Evaluating for Diversity in Question Generation over Text
• MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
• Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation
• Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
• Quiz-Style Question Generation for News Stories