23 papers
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
Zhanbo Hua, Yifan Yao, Weihao Xie +14
While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synt…
WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
Xinping Lei, Xinyu Che, Junqi Xiong +16
Large language models are rapidly evolving into interactive coding agents capable of end-to-end web coding, yet existing benchmarks evaluate only narrow slices of this capability,…
CodeTracer: Towards Traceable Agent States
Han Li, Yifan Yao, Letian Zhu +13
Code agents are advancing rapidly, but debugging them is becoming increasingly difficult. As frameworks orchestrate parallel tool calls and multi-stage workflows over complex tasks…
Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling
Qingyang Xu, Yaling Shen, Stephanie Fong +7
The increasing use of large language models (LLMs) in mental healthcare raises safety concerns in high-stakes therapeutic interactions. A key challenge is distinguishing therapeuti…
NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
Jingzhe Ding, Shengda Long, Changxin Pu +46
Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities re…
Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
Yiming Liang, Yizhi Li, Yantao Du +14
Benchmarks play a crucial role in tracking the rapid advancement of large language models (LLMs) and identifying their capability boundaries. However, existing benchmarks predomina…