Unifying Human and Statistical Evaluation for Natural Language Generation
arXiv:1904.02792
Abstract
How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity but not quality, as models that occasionally emit low quality samples would be insufficiently penalized. In this paper, we propose a unified framework which evaluates both diversity and quality, based on the optimal error rate of predicting whether a sentence is human- or machine-generated. We demonstrate that this error rate can be efficiently estimated by combining human and statistical evaluation, using an evaluation metric which we call HUSE. On summarization and chit-chat dialogue, we show that (i) HUSE detects diversity defects which fool pure human evaluation and that (ii) techniques such as annealing for improving quality actually decrease HUSE due to decreased diversity.
NAACL Camera Ready Submission
References in corpus (13)
- Convolutional Sequence to Sequence Learning
- Exploring the Limits of Language Modeling
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- A note on the evaluation of generative models
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- Adversarial Learning for Neural Dialogue Generation
- A Retrieve-and-Edit Framework for Predicting Structured Outputs
- deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
- Language GANs Falling Short
- Adversarial Evaluation of Dialogue Models
- The price of debiasing automatic metrics in natural language evaluation
- Skill Rating for Generative Models
- Retrieval-Based Neural Code Generation
Cited by in corpus (19)
- Human heuristics for AI-generated language are flawed
- Evaluation of Text Generation: A Survey
- Defending Against Neural Fake News
- Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
- Neural Language Generation: Formulation, Methods, and Evaluation
- Trading Off Diversity and Quality in Natural Language Generation
- Residual Energy-Based Models for Text
- Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
- Content Planning for Neural Story Generation with Aristotelian Rescoring
- Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
- Designing Precise and Robust Dialogue Response Evaluators
- Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
- Approximating Human Judgment of Generated Image Quality
- How to Build User Simulators to Train RL-based Dialog Systems
- How to Evaluate Your Dialogue Models: A Review of Approaches
- UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation
- Language Model Evaluation in Open-ended Text Generation
- Generating Synthetic Text Data to Evaluate Causal Inference Methods
- Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels