4 papers · 1 filter
When Facts Change: Probing LLMs on Evolving Knowledge with evolveQA
Nishanth Sridhar Nakshatri, Shamik Roy, Manoj Ghuhan Arivazhagan +3
LLMs often fail to handle temporal knowledge conflicts--contradictions arising when facts evolve over time within their training data. Existing studies evaluate this phenomenon thr…
When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue
Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou +3
Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and tools. While existing evaluation m…
Constrained Decoding with Speculative Lookaheads
Nishanth Nakshatri, Shamik Roy, Rajarshi Das +3
Constrained decoding with lookahead heuristics (CDLH) is a highly effective method for aligning LLM generations to human preferences. However, the extensive lookahead roll-out oper…
FLAP: Flow-Adhering Planning with Constrained Decoding in LLMs
Shamik Roy, Sailik Sengupta, Daniele Bonadiman +2
Planning is a crucial task for agents in task oriented dialogs (TODs). Human agents typically resolve user issues by following predefined workflows, decomposing workflow steps into…