4 papers
Benchmarking Coding Agents under Evolving Requirements
Jia Feng, Alyssa Wu, Cuiyun Gao +3
Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated g…
DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation
Haoyuan Shi, Mingtao Chen, Shuo Jiang +12
Commercial short-drama production follows a multi-stage chain: script, storyboard, keyframe imagery, shot-level video, and the finished short drama. Most existing benchmarks evalua…
TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
Yunjia Qi, Zehua Yin, Xintong Shi +10
LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to…
Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?
Yangda Peng, Yunjia Qi, Haotian Xia +8
Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric sco…