4 papers
BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests
Zetong Xiong, Qiao Zhao, Jun Zhang +20
Coding-agent benchmarks increasingly cover long-horizon, end-to-end, and interactive development, but typically retain one requested outcome or a fixed change sequence. Sequential…
SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents
Qiao Zhao, JianYing Qu, Jun Zhang +3
Realistic coding-agent benchmarks often replay public GitHub issues and pull requests, making them vulnerable to overlap with model pretraining, fine-tuning, synthetic-data generat…
Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA
Jun Zhang, JianYing Qu, Hanwen Du +3
We present Code-QA-Bench, a fully automated framework for synthesizing repository-level code understanding benchmarks that separates genuine code comprehension from documentation r…
Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity
Doyoung Kim, Zhiwei Ren, Jie Hao +11
We introduce WildAGTEval, a benchmark designed to evaluate large language model (LLM) agents' function-calling capabilities under realistic API complexity. Unlike prior work that a…