4 papers
SAGE: A Service Agent Graph-guided Evaluation Benchmark
Ling Shi, Yuqin Dai, Ziyin Wang +7
The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominan…
DMRetriever: A Family of Models for Improved Text Retrieval in Disaster Management
Kai Yin, Xiangjue Dong, Chengkai Liu +4
Effective and efficient access to relevant information is essential for disaster management. However, no retrieval model is specialized for disaster management, and existing genera…
Z-Scores: A Metric for Linguistically Assessing Disfluency Removal
Maria Teleki, Sai Janjur, Haoran Liu +10
Evaluating disfluency removal in speech requires more than aggregate token-level scores. Traditional word-based metrics such as precision, recall, and F1 (E-Scores) capture overall…
Conversational Speech Reveals Structural Robustness Failures in SpeechLLM Backbones
Maria Teleki, Sai Janjur, Haoran Liu +11
LLMs serve as the backbone in SpeechLLMs, yet their behavior on spontaneous conversational input remains poorly understood. Conversational speech contains pervasive disfluencies --…