Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization

Abstract

Text summarization refers to the process that generates a shorter form of text from the source document preserving salient information. Many existing works for text summarization are generally evaluated by using recall-oriented understudy for gisting evaluation (ROUGE) scores. However, as ROUGE scores are computed based on n-gram overlap, they do not reflect semantic meaning correspondences between generated and reference summaries. Because Korean is an agglutinative language that combines various morphemes into a word that express several meanings, ROUGE is not suitable for Korean summarization. In this paper, we propose evaluation metrics that reflect semantic meanings of a reference summary and the original document, Reference and Document Aware Semantic Score (RDASS). We then propose a method for improving the correlation of the metrics with human judgment. Evaluation results show that the correlation with human judgment is significantly higher for our evaluation metrics than for ROUGE scores.

arXiv commentCOLING 2020

References in corpus (27)

• Teaching Machines to Read and Comprehend
• BERTScore: Evaluating Text Generation with BERT
• Universal Sentence Encoder
• A Deep Reinforced Model for Abstractive Summarization
• A large annotated corpus for learning natural language inference
• Text Summarization with Pretrained Encoders
• ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks
• MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Classify or Select: Neural Architectures for Extractive Document Summarization
• Generating Wikipedia by Summarizing Long Sequences
• End-to-End Retrieval in Continuous Space
• Neural Extractive Summarization with Side Information
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
• Neural Extractive Text Summarization with Syntactic Compression
• A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss
• BanditSum: Extractive Summarization as a Contextual Bandit
• Searching for Effective Neural Extractive Summarization: What Works and What's Next
• KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding
• What comes next? Extractive summarization by next-sentence prediction
• Better Summarization Evaluation with Word Embeddings for ROUGE
• Unsupervised Abstractive Sentence Summarization using Length Controlled Variational Autoencoder
• Concept Pointer Network for Abstractive Summarization
• Self-Supervised Learning for Contextualized Extractive Summarization
• Closed-Book Training to Improve Summarization Encoder Memory
• Extractive Summarization of Long Documents by Combining Global and Local Context