The price of debiasing automatic metrics in natural language evaluation

Abstract

For evaluating generation systems, automatic metrics such as BLEU cost nothing to run but have been shown to correlate poorly with human judgment, leading to systematic bias against certain model improvements. On the other hand, averaging human judgments, the unbiased gold standard, is often too expensive. In this paper, we use control variates to combine automatic metrics with human evaluation to obtain an unbiased estimator with lower cost than human evaluation alone. In practice, however, we obtain only a 7-13% cost reduction on evaluating summarization and open-response question answering systems. We then prove that our estimator is optimal: there is no unbiased estimator with lower cost. Our theory further highlights the two fundamental bottlenecks---the automatic metric and the prompt shown to human evaluators---both of which need to be improved to obtain greater cost savings.

To appear ACL 2018

References in corpus (5)

• A Deep Reinforced Model for Abstractive Summarization
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• Variational Bayesian Inference with Stochastic Search
• S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
• Referenceless Quality Estimation for Natural Language Generation

Cited by in corpus (15)

• BERTScore: Evaluating Text Generation with BERT
• Towards a Human-like Open-Domain Chatbot
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
• Unifying Human and Statistical Evaluation for Natural Language Generation
• Learning to summarize from human feedback
• Neural Language Generation: Formulation, Methods, and Evaluation
• ARAML: A Stable Adversarial Training Framework for Text Generation
• Unsupervised Extractive Summarization using Pointwise Mutual Information
• Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation
• Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
• Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning
• GRUEN for Evaluating Linguistic Quality of Generated Text
• Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation
• Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT
• SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics