CoCon: A Self-Supervised Approach for Controlled Text Generation
arXiv:2006.03535
Abstract
Pretrained Transformer-based language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are studies that seek to control high-level attributes (such as sentiment and topic) of generated text, there is still a lack of more precise control over its content at the word- and phrase-level. Here, we propose Content-Conditioner (CoCon) to control an LM's output text with a content input, at a fine-grained level. In our self-supervised approach, the CoCon block learns to help the LM complete a partially-observed text sequence by conditioning with content inputs that are withheld from the LM. Through experiments, we show that CoCon can naturally incorporate target content into generated texts and control high-level text attributes in a zero-shot manner.
arXiv commentICLR 2021 Camera-Ready
References in corpus (13)
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Language Models are Few-Shot Learners
- The Curious Case of Neural Text Degeneration
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Style Transfer from Non-Parallel Text by Cross-Alignment
- Fine-Tuning Language Models from Human Preferences
- Synthesizer: Rethinking Self-Attention in Transformer Models
- Assessing The Factual Accuracy of Generated Text
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation
- Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer
- Controlling Output Length in Neural Encoder-Decoders
- What makes a good conversation? How controllable attributes affect human judgments
- Simple, Scalable Adaptation for Neural Machine Translation