Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings
arXiv:2104.05156
Abstract
We propose a new reference-free summary quality evaluation measure, with emphasis on the faithfulness. The measure is designed to find and count all possible minute inconsistencies of the summary with respect to the source document. The proposed ESTIME, Estimator of Summary-to-Text Inconsistency by Mismatched Embeddings, correlates with expert scores in summary-level SummEval dataset stronger than other common evaluation measures not only in Consistency but also in Fluency. We also introduce a method of generating subtle factual errors in human summaries. We show that ESTIME is more sensitive to subtle errors than other common evaluation measures.
arXiv comment6 pages, 1 figure, 3 tables
References in corpus (5)
- BERTScore: Evaluating Text Generation with BERT
- GO FIGURE: A Meta Evaluation of Factuality in Summarization
- Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
- Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
- Sensitivity of BLANC to human-scored qualities of text summaries