5 papers
CHI: A Composite Hallucination Index Unifying Entity, Relation, and Quantity Dimensions for Summarization Evaluation
Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala
Faithfulness evaluation of abstractive summaries remains an open challenge, with existing metrics addressing only isolated hallucination types: factual entity errors, relational in…
A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
Praveen Kumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala +1
Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events. Such relation-lev…
Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation
Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala
Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA)…
Bridging the Data Gap: Creating a Hindi Text Summarization Dataset from the English XSUM
Praveenkumar Katwe, RakeshChandra Balabantaray, Kaliprasad Vittala
Current advancements in Natural Language Processing (NLP) have largely favored resource-rich languages, leaving a significant gap in high-quality datasets for low-resource language…
Reducing Hallucinations in Summarization via Reinforcement Learning with Entity Hallucination Index
Praveenkumar Katwe, Rakesh Chandra, Balabantaray Kali +1
Reducing hallucinations in abstractive summarization remains a critical challenge for deploying language models (LMs) in real-world settings. In this work, we introduce a rewarddri…