5 papers
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
Pujun Zheng, Zixin Shang, Shufan Jiang +5
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluat…
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
Kai Chen, Zichen Ding, Jiaye Ge +20
As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly…
GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers
Shufan Jiang, Chios Chen, Zhiyang Chen
The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes b…
NERdME: a Named Entity Recognition Dataset for Indexing Research Artifacts in Code Repositories
Genet Asefa Gesese, Zongxiong Chen, Shufan Jiang +4
Existing scholarly information extraction (SIE) datasets focus on scientific papers and overlook implementation-level details in code repositories. README files describe datasets,…
NFDI4DS Shared Tasks for Scholarly Document Processing
Raia Abu Ahmad, Rana Abdulla, Tilahun Abedissa Taffa +18
Shared tasks are powerful tools for advancing research through community-based standardised evaluation. As such, they play a key role in promoting findable, accessible, interoperab…