4 papers · 1 filter
VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks
Liyang Fan, Yingcheng Shi, Yongbin Li +7
Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluations do not show whether those sy…
SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction
Yuanxiang Shi, Jiayi Lin, Xuanyong Lin +8
Vulnerability discovery is becoming an important ability of large language model (LLM) agents: agents that silently miss real defects leave critical software exposed. Rigorously me…
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
Jialong Chen, Xander Xu, Hu Wei +2
Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing. However, in the real world, the…
V-GameGym: Visual Game Generation for Code Large Language Models
Wei Zhang, Jack Yang, Renshuai Tao +9
Code large language models have demonstrated remarkable capabilities in programming tasks, yet current benchmarks primarily focus on single modality rather than visual game develop…