15 papers
VisualTrust: Video Reasoning under On-Screen Claim Framing
Huachao Zhu, Junjielong Xu, Sicheng Li +5
On-screen text can describe what a video shows, or it can frame what the video is supposed to mean. Current benchmarks treat both alike, testing whether a model can read the text r…
AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses
Kaiyu Huang, Yiting Li, Angyang Gu +3
Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these information gaps remain poorly cha…
CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum
Junjielong Xu, Huachao Zhu, Alyssa Wu +2
The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test…
SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
Zhiqing Zhong, Mingyu Chen, Aoyang Fang +5
As coding agents become more capable, tests remain the primary evidence for judging whether their code is correct. Yet passing available tests does not establish that an implementa…
Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development
Mingyu Chen, Zhiqing Zhong, Manyi Wang +3
As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users often begin with vague intent, lea…
PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
Manyi Wang, Junjielong Xu, Pinjia He
SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired w…
SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
Boxi Yu, Yang Cao, Yuzhong Zhang +9
The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we show that this performance is inflated. Our re-evaluation reveals th…
DeLog: An Efficient Log Compression Framework with Pattern Signature Synthesis
Siyu Yu, Yifan Wu, Junjielong Xu +8
Parser-based log compression, which separates static templates from dynamic variables, is a promising approach to exploit the unique structure of log data. However, its performance…
MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning
Jianbo Yu, Yixuan Li, Hai Xu +5
Log parsing converts semi-structured logs into structured templates, forming a critical foundation for downstream analysis. Traditional syntax and semantic-based parsers often stru…
SWE-Manager: Selecting and Synthesizing Golden Proposals Before Coding
Boyin Tan, Haoning Deng, Junyuan Zhang +3
Large language model (LLM) research in software engineering has largely focused on tasks such as code generation and bug repair. In practice, teams often draft multiple candidate p…
BackportBench: A Multilingual Benchmark for Automated Backporting of Patches
Zhiqing Zhong, Jiaming Huang, Pinjia He
Many modern software projects evolve rapidly to incorporate new features and security patches. It is important for users to update their dependencies to safer versions, but many st…
A Goal-Driven Survey on Root Cause Analysis
Aoyang Fang, Haowen Yang, Haoze Dong +3
Root Cause Analysis (RCA) is a crucial aspect of incident management in large-scale cloud services. While the term root cause analysis or RCA has been widely used, different studie…
Scalable Supervision for Software Agents via Patch Reasoning
Junjielong Xu, Boyin Tan, Xiaoyuan Liu +3
While language model agents have advanced software engineering, existing test-based supervision is limiting its scalability on real-world issues. The reason is twofold: (1) high-co…
Rethinking the Evaluation of Microservice RCA with a Fault Propagation-Aware Benchmark
Aoyang Fang, Songhan Zhang, Yifan Yang +7
While cloud-native microservice architectures have revolutionized software development, their inherent operational complexity makes failure Root Cause Analysis (RCA) a critical yet…
SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints
Zhiyu Fan, Kirill Vasilevski, Dayi Lin +6
The advancement of large language models (LLMs) and code agents has demonstrated significant potential to assist software engineering (SWE) tasks, such as autonomous issue resoluti…