5 papers
Residual Skill Optimization for Text-to-SQL Ensembles
Jiongli Zhu, Haoquan Guan, Parjanya Prajakta Prashant +8
Text-to-SQL ensembles improve over single-candidate generation by drawing multiple SQL candidates and selecting one, but their effectiveness is bounded by Pass@K, the probability t…
SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
Hao Guan, Lingyue Fu, Shao Zhang +8
As autonomous code agents move toward end-to-end software development, evaluating their practical autonomy becomes critical. Current benchmarks hide friction by testing agents in p…
Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation
Lingyue Fu, Bolun Zhang, Hao Guan +7
Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs). However, existing benchmarks for code ag…
On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source
Haoyu Gao, Peerachai Banyongrakkul, Hao Guan +2
Large Language Models (LLMs) increasingly automate software engineering tasks. While recent studies highlight the accelerated adoption of ``AI as a teammate'' in Open Source Softwa…
CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks
Lingyue Fu, Hao Guan, Bolun Zhang +10
The evaluation of Large Language Models (LLMs) for software engineering has shifted towards complex, repository-level tasks. However, existing benchmarks predominantly rely on coar…