14 papers
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
Zhanbo Hua, Yifan Yao, Weihao Xie +14
While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synt…
Knowledge Index of Noah's Ark
Sheng Jin, Minghao Liu, Yunze Xiao +24
Knowledge benchmarks for LLMs face three issues: scaling-driven designs that do not operationalize disciplinary representativeness; flat-payment annotation that permits lazy consen…
Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution
Han Li, Jinyu Tian, Rili Feng +10
Large language models (LLMs) still struggle with the rigorous reasoning demands of hard competitive programming. While recent multi-agent frameworks attempt to bridge this reliabil…
WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
Xinping Lei, Xinyu Che, Junqi Xiong +16
Large language models are rapidly evolving into interactive coding agents capable of end-to-end web coding, yet existing benchmarks evaluate only narrow slices of this capability,…
CodeTracer: Towards Traceable Agent States
Han Li, Yifan Yao, Letian Zhu +13
Code agents are advancing rapidly, but debugging them is becoming increasingly difficult. As frameworks orchestrate parallel tool calls and multi-stage workflows over complex tasks…
SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
Jingxuan Xu, Ken Deng, Weihao Li +36
Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workf…