FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization

Abstract

Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Existing automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated summary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful. Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, which leverages recent advances in reading comprehension. Given question-answer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.

arXiv commentAccepted to ACL 2020

Cited by in corpus (6)

• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics
• Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures
• Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
• Explainable Fact-checking through Question Answering
• Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection