Few-Shot Text Generation with Pattern-Exploiting Training
arXiv:2012.11926
Abstract
Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion. Moreover, when combined with regular learning from examples, this idea yields impressive few-shot results for a wide range of text classification tasks. It is also a promising direction to improve data efficiency in generative settings, but there are several challenges to using a combination of task descriptions and example-based learning for text generation. In particular, it is crucial to find task descriptions that are easy to understand for the pretrained model and to ensure that it actually makes good use of them; furthermore, effective measures against overfitting have to be implemented. In this paper, we show how these challenges can be tackled: We introduce GenPET, a method for text generation that is based on pattern-exploiting training, a recent approach for combining textual instructions with supervised learning that only works for classification tasks. On several summarization and headline generation datasets, GenPET gives consistent improvements over strong baselines in few-shot settings.
arXiv commentAccepted at EMNLP2021
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- Language Models are Few-Shot Learners
- Teaching Machines to Read and Comprehend
- Unsupervised Data Augmentation for Consistency Training
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Measuring and Improving Consistency in Pretrained Language Models
- The Turking Test: Can Language Models Understand Instructions?
Cited by in corpus (7)
- BARTScore: Evaluating Generated Text as Text Generation
- True Few-Shot Learning with Language Models
- Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
- Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation
- Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
- Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?
- Analyzing Commonsense Emergence in Few-shot Knowledge Models