MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

Abstract

A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.

EMNLP19 Camera-Ready

References in corpus (4)

• Teaching Machines to Read and Comprehend
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
• Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations

Cited by in corpus (9)

• Evaluation of Text Generation: A Survey
• Controllable Abstractive Dialogue Summarization with Sketch Supervision
• Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning
• FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation
• OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics
• UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation
• Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation
• Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems
• Vec2Sent: Probing Sentence Embeddings with Natural Language Generation