7 papers
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Chan…
Aspire: Can Models Self-Evolve from Vague Goals?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi +18
Many important forms of human learning begin with a vague goal, such as "become a better physicist" or "improve at research." Learners must interpret the goal, identify capability…
S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
Jiajun Shi, Siyuan Tao, Yuhao Wu +18
Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them…
Repo2Skill-Evo: Repository Skills Go Stale in Silence
Chenyuan Duan, Ge Shi, Zineng Mao +10
Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, w…
StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
Liya Zhu, Xin Ma, Tao Liu +35
Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on re…
Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
Zining Huang, Haoran Que, Hong Zeng +8
When a coding agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks cannot tell the difference: they concentrate rules in t…