Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation

Sanjana Ramprasad, Byron C. Wallace

Modern LLMs can now produce highly readable abstractive summaries, to the point that traditional automated metrics for evaluating summary quality, such as ROUGE, have saturated. Ho…