Sentence-Permuted Paragraph Generation

Abstract

Generating paragraphs of diverse contents is important in many applications. Existing generation models produce similar contents from homogenized contexts due to the fixed left-to-right sentence order. Our idea is permuting the sentence orders to improve the content diversity of multi-sentence paragraph. We propose a novel framework PermGen whose objective is to maximize the expected log-likelihood of output paragraph distributions with respect to all possible sentence orders. PermGen uses hierarchical positional embedding and designs new procedures for training, decoding, and candidate ranking in the sentence-permuted generation. Experiments on three paragraph generation benchmarks demonstrate PermGen generates more diverse outputs with a higher quality than existing models.

arXiv commentEMNLP 2021 (long paper)

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• Non-Autoregressive Neural Machine Translation
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
• A Survey of Knowledge-Enhanced Text Generation
• Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• A Deep Generative Framework for Paraphrase Generation
• Mixture Models for Diverse Machine Translation: Tricks of the Trade
• PaperRobot: Incremental Draft Generation of Scientific Ideas
• Graph-based Multi-hop Reasoning for Long Text Generation