10 papers · 1 filter
Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development
Mingyu Chen, Zhiqing Zhong, Manyi Wang +3
As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, lea…
PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
Manyi Wang, Junjielong Xu, Pinjia He
SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired w…
SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
Boxi Yu, Yang Cao, Yuzhong Zhang +9
The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we show that this performance is inflated. Our re-evaluation reveals th…
DeLog: An Efficient Log Compression Framework with Pattern Signature Synthesis
Siyu Yu, Yifan Wu, Junjielong Xu +8
Parser-based log compression, which separates static templates from dynamic variables, is a promising approach to exploit the unique structure of log data. However, its performance…
MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning
Jianbo Yu, Yixuan Li, Hai Xu +5
Log parsing converts semi-structured logs into structured templates, forming a critical foundation for downstream analysis. Traditional syntax and semantic-based parsers often stru…
SWE-Manager: Selecting and Synthesizing Golden Proposals Before Coding
Boyin Tan, Haoning Deng, Junyuan Zhang +3
Large language model (LLM) research in software engineering has largely focused on tasks such as code generation and bug repair. In practice, teams often draft multiple candidate p…
BackportBench: A Multilingual Benchmark for Automated Backporting of Patches
Zhiqing Zhong, Jiaming Huang, Pinjia He
Many modern software projects evolve rapidly to incorporate new features and security patches. It is important for users to update their dependencies to safer versions, but many st…
A Goal-Driven Survey on Root Cause Analysis
Aoyang Fang, Haowen Yang, Haoze Dong +3
Root Cause Analysis (RCA) is a crucial aspect of incident management in large-scale cloud services. While the term root cause analysis or RCA has been widely used, different studie…
Rethinking the Evaluation of Microservice RCA with a Fault Propagation-Aware Benchmark
Aoyang Fang, Songhan Zhang, Yifan Yang +7
While cloud-native microservice architectures have revolutionized software development, their inherent operational complexity makes failure Root Cause Analysis (RCA) a critical yet…
SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints
Zhiyu Fan, Kirill Vasilevski, Dayi Lin +6
The advancement of large language models (LLMs) and code agents has demonstrated significant potential to assist software engineering (SWE) tasks, such as autonomous issue resoluti…