Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation

Abstract

Sequence-to-Sequence (S2S) neural text generation models, especially the pre-trained ones (e.g., BART and T5), have exhibited compelling performance on various natural language generation tasks. However, the black-box nature of these models limits their application in tasks where specific rules (e.g., controllable constraints, prior knowledge) need to be executed. Previous works either design specific model structure (e.g., Copy Mechanism corresponding to the rule "the generated output should include certain words in the source input") or implement specialized inference algorithm (e.g., Constrained Beam Search) to execute particular rules through the text generation. These methods require careful design case-by-case and are difficult to support multiple rules concurrently. In this paper, we propose a novel module named Neural Rule-Execution Tracking Machine that can be equipped into various transformer-based generators to leverage multiple rules simultaneously to guide the neural generation model for superior generation performance in a unified and scalable way. Extensive experimental results on several benchmarks verify the effectiveness of our proposed model in both controllable and general text generation.

References in corpus (17)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Fine-Tuning Language Models from Human Preferences
• Modeling Coverage for Neural Machine Translation
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Neural Machine Translation with External Phrase Memory
• Minimum Risk Training for Neural Machine Translation
• Controlling Output Length in Neural Encoder-Decoders
• A Distributional Approach to Controlled Text Generation
• KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning
• CoCon: A Self-Supervised Approach for Controlled Text Generation
• TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
• Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization
• NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints
• Improving Neural Machine Translation through Phrase-based Forced Decoding