4 papers · 1 filter
AdaGuard: Enhancing Safety and Policy Compliance with Reasoning-Enabled LLM-As-A-Judge Guardrails
Melissa Kazemi Rad, Sihui Dai, Isha Slavin +6
Enterprise generative AI applications require robust safety mechanisms that can accommodate diverse risk postures, evolving policies, and varying latency constraints. Current guard…
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Sangwoo Cho, Kushal Chawla, Pengshan Cai +4
Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, an…
SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation
Chenyang Zhu, Jiayu Yao, Kushal Chawla +10
As autonomous agents tackle increasingly complex multi-step, multi-agent tasks, their execution trajectories have scaled beyond the constraints of even the largest context windows.…
RAFFLES: Reasoning-based Attribution of Faults for LLM Systems
Chenyang Zhu, Spencer Hong, Jingyu Wu +6
The advent of complex, interconnected long-horizon LLM systems has made it incredibly tricky to identify where and when these systems break down. Evaluation capabilities that curre…