ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

Abstract

Current pre-training works in natural language generation pay little attention to the problem of exposure bias on downstream tasks. To address this issue, we propose an enhanced multi-flow sequence to sequence pre-training and fine-tuning framework named ERNIE-GEN, which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder. Experimental results demonstrate that ERNIE-GEN achieves state-of-the-art results with a much smaller amount of pre-training data and parameters on a range of language generation tasks, including abstractive summarization (Gigaword and CNN/DailyMail), question generation (SQuAD), dialogue generation (Persona-Chat) and generative question answering (CoQA).

arXiv commentThe source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE. We have also updated the performances of ERNIE-GEN under a larger scaled pre-training corpora in appendix A

References in corpus (5)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Teaching Machines to Read and Comprehend
• ERNIE: Enhanced Representation through Knowledge Integration
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks

Cited by in corpus (10)

• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
• On the Effectiveness of Pretrained Models for API Learning
• Accelerating Real-Time Question Answering via Question Generation
• Contrastive Learning with Adversarial Perturbations for Conditional Text Generation
• Exploring LLMs for Automated Generation and Adaptation of Questionnaires
• Enhancing Question Generation with Commonsense Knowledge
• Resurrecting Submodularity for Neural Text Generation
• Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance
• Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation