6 papers · 1 filter
SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
Pengyu Xue, He Yang Yuan, Xin Wang +6
Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains…
To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair
Zhihao Lin, Junhua Zhu, Mingyi Zhou +5
LLM-based agents for program repair are increasingly built on a "generate-run-revise" paradigm, iteratively executing tests to evaluate and refine patches. This execution-based app…
Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
Xin Wang, Liangtai Sun, Yaoming Zhu +8
Existing code-generation benchmarks score a single mapping from a complete prompt to a one-shot output. However, real web development is different. Users seldom write a full spec a…
Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs
He Yang Yuan, Xin Wang, Kundi Yao +3
Logging code plays an important role in software systems by recording key events and behaviors, which are essential for debugging and monitoring. However, insecure logging practice…
LLM4Perf: Large Language Models Are Effective Samplers for Multi-Objective Performance Modeling
Xin Wang, Zhenhao Li, Zishuo Ding
The performance of modern software systems is critically dependent on their complex configuration options. Building accurate performance models to navigate this vast space requires…
Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning
Xin Wang, Zhenhao Li, Zishuo Ding
Logging code is written by developers to capture system runtime behavior and plays a vital role in debugging, performance analysis, and system monitoring. However, defects in loggi…