Unified Language Model Pre-training for Natural Language Understanding and Generation

Abstract

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm.

Accepted by NeurIPS-19. Code and pre-trained models: https://github.com/microsoft/unilm

References in corpus (11)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Gaussian Error Linear Units (GELUs)
• A Deep Reinforced Model for Abstractive Summarization
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Semi-supervised Sequence Learning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Learning and Evaluating General Linguistic Intelligence
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Pre-trained Language Model Representations for Language Generation
• Read + Verify: Machine Reading Comprehension with Unanswerable Questions

Cited by in corpus (81)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Pre-trained Models for Natural Language Processing: A Survey
• Multilingual Denoising Pre-training for Neural Machine Translation
• A Survey of Knowledge-Enhanced Text Generation
• Multi-Task Deep Neural Networks for Natural Language Understanding
• StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement
• Text Summarization with Pretrained Encoders
• Pretrained Language Models for Sequential Sentence Classification
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognition
• Adversarial Training for Large Neural Language Models
• Pre-training via Paraphrasing
• Representation Learning for Natural Language Processing
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
• Abstractive Text Summarization: State of the Art, Challenges, and Improvements
• Towards an Online Empathetic Chatbot with Emotion Causes
• Text Compression-aided Transformer Encoding
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis
• Multi-hop Question Generation with Graph Convolutional Network
• Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
• CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection
• UP-DETR: Unsupervised Pre-training for Object Detection with Transformers
• CO-Search: COVID-19 Information Retrieval with Semantic Search, Question Answering, and Abstractive Summarization
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• RepsNet: Combining Vision with Language for Automated Medical Reports
• Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
• Explainable Natural Language Processing for Corporate Sustainability Analysis
• Self-Training Vision Language BERTs with a Unified Conditional Model
• A Survey of Knowledge Enhanced Pre-trained Models
• Encoder-Agnostic Adaptation for Conditional Language Generation
• Visualizing and Understanding the Effectiveness of BERT
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• Improving Readability for Automatic Speech Recognition Transcription
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
• DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
• Zero-Shot Paraphrase Generation with Multilingual Language Models
• Ad Headline Generation using Self-Critical Masked Language Model
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks
• TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
• Convolution-enhanced Evolving Attention Networks
• Explainable Recommendation with Personalized Review Retrieval and Aspect Learning
• Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
• Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
• Tree Transformer: Integrating Tree Structures into Self-Attention
• Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
• Defending Against Backdoor Attacks in Natural Language Generation
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Unified Question Generation with Continual Lifelong Learning
• Mixed-Lingual Pre-training for Cross-lingual Summarization
• Towards Efficiently Evaluating the Robustness of Deep Neural Networks in IoT Systems: A GAN-based Method
• Discriminative Adversarial Search for Abstractive Summarization
• Improving Non-autoregressive Generation with Mixup Training
• Leveraging Lead Bias for Zero-shot Abstractive News Summarization
• Diverse Pretrained Context Encodings Improve Document Translation
• A Closer Look at Data Bias in Neural Extractive Summarization Models
• Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
• Pre-training Text Representations as Meta Learning
• On the Trade-off between Redundancy and Local Coherence in Summarization
• Denoising based Sequence-to-Sequence Pre-training for Text Generation
• Composed Variational Natural Language Generation for Few-shot Intents
• MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization
• From Large to Tiny: Distilling and Refining Mathematical Expertise for Math Word Problems with Weakly Supervision
• ZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generation
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
• Explicit Memory Tracker with Coarse-to-Fine Reasoning for Conversational Machine Reading
• Improved Synthetic Training for Reading Comprehension
• Which Discriminator for Cooperative Text Generation?
• Improving Factual Consistency of Abstractive Summarization on Customer Feedback
• Global Context Networks
• Mathematical Word Problem Generation from Commonsense Knowledge Graph and Equations
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data
• Boosting Summarization with Normalizing Flows and Aggressive Training
• Pseudo Siamese Network for Few-shot Intent Generation
• Improving Unsupervised Question Answering via Summarization-Informed Question Generation
• Small-Bench NLP: Benchmark for small single GPU trained models in Natural Language Processing
• Explaining Documents' Relevance to Search Queries
• Autoregressive Knowledge Distillation through Imitation Learning
• Simple and Effective Input Reformulations for Translation