An Empirical Study of Extrapolation in Text Generation with Scalar Control
arXiv:2104.07910
Abstract
We conduct an empirical evaluation of extrapolation performance when conditioning on scalar control inputs like desired output length, desired edit from an input sentence, and desired sentiment across three text generation tasks. Specifically, we examine a zero-shot setting where models are asked to generalize to ranges of control values not seen during training. We focus on evaluating popular embedding methods for scalar inputs, including both learnable and sinusoidal embeddings, as well as simpler approaches. Surprisingly, our findings indicate that the simplest strategy of using scalar inputs directly, without further encoding, most reliably allows for successful extrapolation.
References in corpus (8)
- A large annotated corpus for learning natural language inference
- A Persona-Based Neural Conversation Model
- Toward Controlled Generation of Text
- Controlling Output Length in Neural Encoder-Decoders
- Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer
- Controlling the Output Length of Neural Machine Translation
- Learning Sparse Prototypes for Text Generation
- Positional Encoding to Control Output Sequence Length