What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
arXiv:2305.11707
Abstract
In Natural Language Generation (NLG) tasks, for any input, multiple communicative goals are plausible, and any goal can be put into words, or produced, in multiple ways. We characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. We then inspect the space of output strings shaped by a generation system's predicted probability distribution and decoding algorithm to probe its uncertainty. For each test input, we measure the generator's calibration to human production variability. Following this instance-level approach, we analyse NLG models and decoding strategies, demonstrating that probing a generator with multiple samples and, when possible, multiple references, provides the level of detail necessary to gain understanding of a model's representation of uncertainty. Code available at https://github.com/dmg-illc/nlg-uncertainty-probes.
Camera ready version for EMNLP 2023
References in corpus (10)
- Scaling Instruction-Finetuned Language Models
- DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
- BARTScore: Evaluating Generated Text as Text Generation
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- Calibrating Sequence likelihood Improves Conditional Language Generation
- Momentum Calibration for Text Generation
- Uncertainty in Natural Language Generation: From Theory to Applications
- Pragmatically Appropriate Diversity for Dialogue Evaluation
- Statistical Model Criticism of Variational Auto-Encoders
- On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation