12 papers
Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
Xiangxin Zhao, Han Li, Shuaiting Li +4
Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a gro…
Independent Test Generation for RESTful APIs
S M Sadrul Islam Asif, James Chen, Kennett Puerto Diaz +2
Modern REST API testing relies on brittle sequences of calls to build system state. These multi-step tests suffer from non-determinism, poor scalability, and a "reachability tax" w…
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 ter…
Hallucination Inspector: A Fact-Checking Judge for API Migration
Marcos Tileria, Santanu Kumar Dash, Profir-Petru Pârţachi +1
Large Language Models (LLMs) are increasingly deployed in automated software engineering for tasks such as API migration. While LLMs are able to identify migration patterns, they o…
Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs
Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi +3
The lack of transparency about code datasets used to train large language models (LLMs) makes it difficult to detect, evaluate, and mitigate data leakage. We present a perturbation…
Compressing Code Context for LLM-based Issue Resolution
Haoxiang Jia, Earl T. Barr, Sergey Mechtaev
Large Language Models (LLMs) are now capable of resolving real-world GitHub issues. However, current approaches overapproximate the code context and suffer from two compounding pro…