5 papers
DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models
Adarsh Kumarappan, Pareesa Ameneh Golnari, Wen Wen +5
DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six pro…
SemRep: Generative Code Representation Learning with Code Transformations
Weichen Li, Jiamin Song, Bogdan Alexandru Stoica +4
Code transformation is a foundational capability in the software development process, where its effectiveness relies on constructing a high-quality code representation to character…
Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
Daoan Zhang, Shuo Zhang, Zijian Jin +3
Pull request (PR) review is essential for ensuring software quality, yet automating this task remains challenging due to noisy supervision, limited contextual understanding, and in…
Lita: Light Agent Uncovers the Agentic Coding Capabilities of LLMs
Hankun Dai, Maoquan Wang, Mengnan Qi +6
Large language models (LLMs) are increasingly being applied to programming tasks, ranging from single-turn code completion to autonomous agents. Current code agent designs frequent…
SWE-bench Goes Live!
Linghao Zhang, Shilin He, Chaoyun Zhang +12
The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs…