SAUCE: Summary Analysis Using Conversation Entailment

Abstract

With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.

arXiv commentAccepted and published to the ACL GEM workshop. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pages 364-377, San Diego, California, USA. Association for Computational Linguistics

References in corpus (10)

• BERTScore: Evaluating Text Generation with BERT
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• No Language Left Behind: Scaling Human-Centered Machine Translation
• Mistral 7B
• Qwen2.5-Coder Technical Report
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges
• Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation
• Advancing Speech Translation: A Corpus of Mandarin-English Conversational Telephone Speech
• Cross-Lingual Conversational Speech Summarization with Large Language Models