Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
arXiv:1901.00398
Abstract
We conduct a large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the context of generating online product reviews. We compare human-based evaluators with a variety of automated evaluation procedures, including discriminative evaluators that measure how well machine-generated text can be distinguished from human-written text, as well as word overlap metrics that assess how similar the generated text compares to human-written references. We determine to what extent these different evaluators agree on the ranking of a dozen of state-of-the-art generators for online product reviews. We find that human evaluators do not correlate well with discriminative evaluators, leaving a bigger question of whether adversarial accuracy is the correct objective for natural language generation. In general, distinguishing machine-generated text is challenging even for human evaluators, and human decisions correlate better with lexical overlaps. We find lexical diversity an intriguing metric that is indicative of the assessments of different evaluators. A post-experiment survey of participants provides insights into how to evaluate and improve the quality of natural language generation systems.
References in corpus (35)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Categorical Reparameterization with Gumbel-Softmax
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- NIPS 2016 Tutorial: Generative Adversarial Networks
- Convolutional Neural Networks for Sentence Classification
- Exploring the Limits of Language Modeling
- Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
- Finding Deceptive Opinion Spam by Any Stretch of the Imagination
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
- Semantic Segmentation using Adversarial Networks
- Unsupervised Cross-Domain Image Generation
- MaskGAN: Better Text Generation via Filling in the______
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Neural Photo Editing with Introspective Adversarial Networks
- How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
- Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
- Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
- Adversarial Ranking for Language Generation
- Adversarial Feature Matching for Text Generation
- Adversarial Generation of Natural Language
- Deep Photo Style Transfer
- Language GANs Falling Short
- On Accurate Evaluation of GANs for Language Generation
- Context-aware Natural Language Generation with Recurrent Neural Networks
- Adversarial Evaluation of Dialogue Models
- Neural Text Generation: Past, Present and Beyond
- Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network
- Hierarchical Text Generation and Planning for Strategic Dialogue
- Data Generation as Sequential Decision Making
- Unifying Human and Statistical Evaluation for Natural Language Generation
- Eval all, trust a few, do wrong to none: Comparing sentence generation models
- Evaluating Text GANs as Language Models
- Jointly Measuring Diversity and Quality in Text Generation Models