4 papers
CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
Jun Gao, Yun Peng, Qian Qiao +6
Despite strong performance on code generation tasks, it remains unclear whether large language models (LLMs) genuinely reason about code execution. Existing code reasoning benchmar…
Environment-Aware Code Generation: How far are We?
Tongtong Wu, Rongyi Chen, Wenjie Du +6
Recent progress in large language models (LLMs) has improved code generation, but most evaluations still test isolated, small-scale code (e.g., a single function) under default or…
LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead
Junda He, Jieke Shi, Terry Yue Zhuo +5
The rapid integration of Large Language Models (LLMs) into software engineering (SE) has revolutionized tasks like code generation, producing a massive volume of software artifacts…
A Benchmark for Localizing Code and Non-Code Issues in Software Projects
Zejun Zhang, Jian Wang, Qingyun Yang +7
Accurate project localization (e.g., files and functions) for issue resolution is a critical first step in software maintenance. However, existing benchmarks for issue localization…