POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

Abstract

Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learning and free-form text generation. However, these models cannot be directly employed to generate text under specified lexical constraints. To address this challenge, we present POINTER (PrOgressive INsertion-based TransformER), a simple yet novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. This procedure is recursively applied until a sequence is completed. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable. We pre-train our model with the proposed progressive insertion-based objective on a 12GB Wikipedia dataset, and fine-tune it on downstream hard-constrained generation tasks. Non-autoregressive decoding yields an empirically logarithmic time complexity during inference time. Experimental results on both News and Yelp datasets demonstrate that POINTER achieves state-of-the-art performance on constrained text generation. We released the pre-trained models and the source code to facilitate future research (https://github.com/dreasysnail/POINTER).

EMNLP 2020 long paper

References in corpus (18)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Towards a Human-like Open-Domain Chatbot
• Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Non-Autoregressive Machine Translation with Disentangled Context Transformer
• KERMIT: Generative Insertion-Based Modeling for Sequences
• Non-Monotonic Sequential Text Generation
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Few-shot Natural Language Generation for Task-Oriented Dialog
• Distilling Knowledge Learned in BERT for Text Generation
• FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
• Enabling Language Models to Fill in the Blanks
• Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!

Cited by in corpus (6)

• Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation
• Neural Language Generation: Formulation, Methods, and Evaluation
• Contextualized Perturbation for Textual Adversarial Attack
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• Cascaded Text Generation with Markov Transformers
• Conditional Text Generation for Harmonious Human-Machine Interaction