Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
arXiv:2004.04228
Abstract
Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose an automatic evaluation protocol called QAGS (pronounced "kags") that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.
arXiv commentACL 2020
References in corpus (9)
- Decoupled Weight Decay Regularization
- Teaching Machines to Read and Comprehend
- Enhanced LSTM for Natural Language Inference
- The Curious Case of Neural Text Degeneration
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Faithful to the Original: Fact Aware Neural Abstractive Summarization
- Assessing The Factual Accuracy of Generated Text
Cited by in corpus (14)
- Evaluation of Text Generation: A Survey
- Recursively Summarizing Books with Human Feedback
- The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
- QuestEval: Summarization Asks for Fact-based Evaluation
- Liputan6: A Large-scale Indonesian Dataset for Text Summarization
- Factual Error Correction for Abstractive Summarization Models
- Zero-shot Fact Verification by Claim Generation
- Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View
- Looking Beyond Sentence-Level Natural Language Inference for Downstream Tasks
- Understanding Points of Correspondence between Sentences for Abstractive Summarization
- Toward Improving Coherence and Diversity of Slogan Generation
- Learning to Fuse Sentences with Transformers for Summarization
- Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey
- Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation