4 papers · 1 filter
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Youbing Yin, Jingyu Wu, Anoop Kumar +3
LLM-as-a-Judge systems are increasingly employed for automated evaluation, yet remain susceptible to systematic biases that undermine their reliability. We demonstrate that the jud…
Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL
Youbing Yin, Xujun Peng, Muhammad S. Uddin +3
Text-to-SQL systems typically assume a one-to-one mapping between natural-language queries and correct SQL translations. In practice, however, many queries are inherently ambiguous…
SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation
Chenyang Zhu, Jiayu Yao, Kushal Chawla +10
As autonomous agents tackle increasingly complex multi-step, multi-agent tasks, their execution trajectories have scaled beyond the constraints of even the largest context windows.…
RAFFLES: Reasoning-based Attribution of Faults for LLM Systems
Chenyang Zhu, Spencer Hong, Jingyu Wu +6
The advent of complex, interconnected long-horizon LLM systems has made it incredibly tricky to identify where and when these systems break down. Evaluation capabilities that curre…