10 papers
When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems
Shihao Weng, Longtian Wang, Yang Feng +6
Software quality has long rested on specifications that state what code must do, and on testing and verification that check that the code does it. Agentic systems change this found…
OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies
Zhaoyang Chu, Earl T. Barr, Claire Le Goues +4
Coding agents are extending their reach into the physical world by writing and executing robot control programs. One might expect the agents to use the existing mature software sta…
LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation
Han Li, Zhemin Fang, Rili Feng +8
Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing ben…
Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
Xiangxin Zhao, Han Li, Shuaiting Li +4
Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a gro…
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 ter…
An Iterative Test-and-Repair Framework for Competitive Code Generation
Lingxiao Tang, Muyang Ye, Zhaoyang Chu +4
Large language models (LLMs) have made remarkable progress in code generation, but competitive programming remains a challenge. Recent training-based methods have improved code gen…
HerAgent: Rethinking the Automated Environment Deployment via Hierarchical Test Pyramid
Xiang Li, Siyu Lu, Federica Sarro +2
Automated software environment setup is a prerequisite for testing, debugging, and reproducing failures, yet remains challenging in practice due to complex dependencies, heterogene…
ContextBench: A Benchmark for Context Retrieval in Coding Agents
Han Li, Letian Zhu, Bohan Zhang +7
LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insi…
Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agents
Xiang Li, Zhiwei Fei, Ying Ma +3
Modern software systems continuously undergo code upgrades to enhance functionality, security, and performance, and Large Language Models (LLMs) have demonstrated remarkable capabi…
JMigBench: A Benchmark for Evaluating LLMs on Source Code Migration (Java 8 to Java 11)
Nishil Amin, Zhiwei Fei, Xiang Li +2
We build a benchmark to evaluate large language models (LLMs) for source code migration tasks, specifically upgrading functions from Java 8 to Java 11. We first collected a dataset…