Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
arXiv:1909.01610
Abstract
Abstractive summarization approaches based on Reinforcement Learning (RL) have recently been proposed to overcome classical likelihood maximization. RL enables to consider complex, possibly non-differentiable, metrics that globally assess the quality and relevance of the generated outputs. ROUGE, the most used summarization metric, is known to suffer from bias towards lexical similarity as well as from suboptimal accounting for fluency and readability of the generated abstracts. We thus explore and propose alternative evaluation measures: the reported human-evaluation analysis shows that the proposed metrics, based on Question Answering, favorably compares to ROUGE -- with the additional property of not requiring reference summaries. Training a RL-based model on these metrics leads to improvements (both in terms of human or automated metrics) over current approaches that use ROUGE as a reward.
arXiv commentAccepted at EMNLP 2019
References in corpus (6)
- Sequence to Sequence Learning with Neural Networks
- Teaching Machines to Read and Comprehend
- A Deep Reinforced Model for Abstractive Summarization
- Sequence Level Training with Recurrent Neural Networks
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- Actor-Critic based Training Framework for Abstractive Summarization
Cited by in corpus (6)
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Recursively Summarizing Books with Human Feedback
- QuestEval: Summarization Asks for Fact-based Evaluation
- Rethinking Automatic Evaluation in Sentence Simplification
- Discriminative Adversarial Search for Abstractive Summarization
- Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View