4 papers · 1 filter
When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue
Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou +3
Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and tools. While existing evaluation m…
When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA
Nishanth Sridhar Nakshatri, Shamik Roy, Manoj Ghuhan Arivazhagan +3
LLMs often fail to handle temporal knowledge conflicts--contradictions arising when facts evolve over time within their training data. Existing studies evaluate this phenomenon thr…
Constrained Decoding with Speculative Lookaheads
Nishanth Nakshatri, Shamik Roy, Rajarshi Das +3
Constrained decoding with lookahead heuristics (CDLH) is a highly effective method for aligning LLM generations to human preferences. However, the extensive lookahead roll-out oper…
FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs
Shamik Roy, Sailik Sengupta, Daniele Bonadiman +2
Planning is a crucial task for agents in task oriented dialogs (TODs). Human agents typically resolve user issues by following predefined workflows, decomposing workflow steps into…