4 papers
OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies
Zhaoyang Chu, Earl T. Barr, Claire Le Goues +4
Coding agents are extending their reach into the physical world by writing and executing robot control programs. One might expect the agents to use the existing mature software sta…
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 ter…
ContextBench: A Benchmark for Context Retrieval in Coding Agents
Han Li, Letian Zhu, Bohan Zhang +7
LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insi…
Prometheus: Towards Long-Horizon Codebase Navigation for Repository-Level Problem Solving
Yue Pan, Zimin Chen, Siyu Lu +8
Large Language Models (LLMs) have shown remarkable capabilities in automating software engineering tasks, spurring the emergence of coding agents that scaffold LLMs with external t…