4 papers
SAGE: A Service Agent Graph-guided Evaluation Benchmark
Ling Shi, Yuqin Dai, Ziyin Wang +7
The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominan…
Conversational Speech Reveals Structural Robustness Failures in SpeechLLM Backbones
Maria Teleki, Sai Janjur, Haoran Liu +11
LLMs serve as the backbone in SpeechLLMs, yet their behavior on spontaneous conversational input remains poorly understood. Conversational speech contains pervasive disfluencies --…
DMRetriever: A Family of Models for Improved Text Retrieval in Disaster Management
Kai Yin, Xiangjue Dong, Chengkai Liu +4
Effective and efficient access to relevant information is essential for disaster management. However, no retrieval model is specialized for disaster management, and existing genera…
Z-Scores: A Metric for Linguistically Assessing Disfluency Removal
Maria Teleki, Sai Janjur, Haoran Liu +10
Evaluating disfluency removal in speech requires more than aggregate token-level scores. Traditional word-based metrics such as precision, recall, and F1 (E-Scores) capture overall…