Asking and Answering Questions to Evaluate the Factual Consistency of Summaries

Abstract

Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose an automatic evaluation protocol called QAGS (pronounced "kags") that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.

ACL 2020

References in corpus (3)

• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Faithful to the Original: Fact Aware Neural Abstractive Summarization

Cited by in corpus (10)

• Recursively Summarizing Books with Human Feedback
• The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
• QuestEval: Summarization Asks for Fact-based Evaluation
• Liputan6: A Large-scale Indonesian Dataset for Text Summarization
• Zero-shot Fact Verification by Claim Generation
• Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View
• Looking Beyond Sentence-Level Natural Language Inference for Downstream Tasks
• Understanding Points of Correspondence between Sentences for Abstractive Summarization
• Toward Improving Coherence and Diversity of Slogan Generation
• Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation