PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation

Abstract

Self-supervised pre-training, such as BERT, MASS and BART, has emerged as a powerful technique for natural language understanding and generation. Existing pre-training techniques employ autoencoding and/or autoregressive objectives to train Transformer-based models by recovering original word tokens from corrupted text with some masked tokens. The training goals of existing techniques are often inconsistent with the goals of many language generation tasks, such as generative question answering and conversational response generation, for producing new text given context. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. The new scheme alleviates the mismatch introduced by the existing denoising scheme between pre-training and fine-tuning where generation is more than reconstructing original text. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.

Accepted at EMNLP 2020

References in corpus (17)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Gaussian Error Linear Units (GELUs)
• Language Models are Few-Shot Learners
• Teaching Machines to Read and Comprehend
• A Neural Conversational Model
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Semi-supervised Sequence Learning
• Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
• Bottom-Up Abstractive Summarization
• ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation
• Global Encoding for Abstractive Summarization
• Incorporating External Knowledge into Machine Reading for Generative Question Answering

Cited by in corpus (5)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• Pre-Trained Models: Past, Present and Future
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Generating Code with the Help of Retrieved Template Functions and Stack Overflow Answers