13 papers
Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code
Alberick Euraste Djire, Iyiola E. Olatunji, Melissa Tessa +3
LLMs are increasingly used for code generation, yet they frequently hallucinate non-existent software packages, creating exploitable entry points into the software supply chain. We…
Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
Xiangxin Zhao, Han Li, Shuaiting Li +4
Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a gro…
Independent Test Generation for RESTful APIs
S M Sadrul Islam Asif, James Chen, Kennett Puerto Diaz +2
Modern REST API testing relies on brittle sequences of calls to build system state. These multi-step tests suffer from non-determinism, poor scalability, and a "reachability tax" w…
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 ter…
Hallucination Inspector: A Fact-Checking Judge for API Migration
Marcos Tileria, Santanu Kumar Dash, Profir-Petru Pârţachi +1
Large Language Models (LLMs) are increasingly deployed in automated software engineering for tasks such as API migration. While LLMs are able to identify migration patterns, they o…
Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs
Djiré Albérick Euraste, Kaboré Abdoul Kader, Jordan Samhi +3
The lack of transparency about code datasets used to train large language models (LLMs) makes it difficult to detect, evaluate, and mitigate data leakage. We present a perturbation…