11 papers
Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance
Chao Peng, Zhiheng Lyu, Peijie Dong +2
The paper proposes a benchmark metric called the horizon residual to compare long-horizon task success against predictions from short-stage baselines, highlighting how performance…
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction
Tencent WorkBuddy Bench Team, Siqi Cai, Shaopeng Chen +35
We introduce Tencent WorkBuddy Bench, a multi-domain evaluation suite for coding agents; this report documents its construction methodology, scoring protocol, and a cross-model lea…
VisCoder2: Building Multi-Language Visualization Coding Agents
Yuansheng Ni, Songcheng Cai, Xiangchao Chen +8
Large language models (LLMs) have recently enabled coding agents capable of generating, executing, and revising visualization code. However, existing models often fail in practical…
StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
Jialin Yang, Dongfu Jiang, Lipeng He +17
As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce Struct…
SWE-Next: Scalable Real-World Software Engineering Tasks for Agents
Jiarong Liang, Zhiheng Lyu, Zijie Liu +4
Executable software engineering data is valuable for training SWE agents, but scaling it remains difficult for two reasons: only a small fraction of real repository changes yield v…
SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding
Songcheng Cai, Zhiheng Lyu, Yuansheng Ni +13
Agentic repository-level code understanding is essential for automating complex software engineering tasks, yet the field lacks reliable benchmarks. Existing evaluations often over…