9 papers
xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems
Yongchang Peng, Qingshui Gu, Liya Zhu +31
Large language models (LLMs) are increasingly used for everyday assistance, yet existing benchmarks only partially reflect the requests users naturally make in practice. Real-world…
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Chan…
Aspire: Can Models Self-Evolve from Vague Goals?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi +18
Many important forms of human learning begin with a vague goal, such as "become a better physicist" or "improve at research." Learners must interpret the goal, identify capability…
S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
Jiajun Shi, Siyuan Tao, Yuhao Wu +18
Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them…
Repo2Skill-Evo: Repository Skills Go Stale in Silence
Chenyuan Duan, Ge Shi, Zineng Mao +10
Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, w…
StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
Liya Zhu, Xin Ma, Tao Liu +35
Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on re…
Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
Zining Huang, Haoran Que, Hong Zeng +8
When a coding agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks cannot tell the difference: they concentrate rules in t…
Knowledge Index of Noah's Ark
Sheng Jin, Minghao Liu, Yunze Xiao +24
Knowledge benchmarks for LLMs face three issues: scaling-driven designs that do not operationalize disciplinary representativeness; flat-payment annotation that permits lazy consen…
Comparing baseball players across eras via novel Full House Modeling
Shen Yan, Adrian Burgos, Christopher Kinson +1
A new methodological framework suitable for era-adjusting baseball statistics is developed in this article. Within this methodological framework specific models are motivated. We c…