6 papers
Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills
Md Tahmid Rahman Laskar, Gundeep Singh, Xue-Yong Fu +11
Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this problem through the lens of Skill LLMs:…
Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators
Riqiang Wang, Elena Khasanova, Harsh Saini +4
As voice agents gain more popularity commercially, the user simulators used to evaluate the deployed agents are also being developed to include more realistic, variable, and divers…
Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation
Lex Konnelly, Elena Khasanova, Riqiang Wang +3
As agentic systems gain commercial popularity, user simulators increasingly serve as measurement instrument for their evaluation. However, the fidelity of simulated users in compar…
When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success
Md Tahmid Rahman Laskar, Xue-Yong Fu, Gundeep Singh +11
Agent models are frequently evaluated one decision at a time, where the model predicts the next action based on the gold interaction history, which is scored against a reference. W…
DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations
Elena Khasanova, Harsh Saini, Md Tahmid Rahman Laskar +3
The rapid advancements in Large Language Models (LLMs) have enabled their adoption in real-world industrial scenarios for various natural language processing tasks. However, the hi…
DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization
Xue-Yong Fu, Elena Khasanova, Md Tahmid Rahman Laskar +2
Large language models (LLMs) have achieved impressive performance in text summarization, yet their performance often falls short when applied to specialized domains that differ fro…