14 papers
SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information
Junjie Ye, Zhuohui Sheng, Shaofan Liu +12
Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps)…
SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
Yujiong Shen, Yajie Yang, Zhiheng Xi +17
Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents' ability to orches…
LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening
Ming Zhang, Qiyuan Peng, Yinxi Wei +13
Evaluating large language models (LLMs) on natural-language logical reasoning is essential because rule-governed tasks require conclusions to follow strictly from stated premises.…
Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies
Ming Zhang, Jiabao Zhuang, Wenqing Jing +18
Deep Research Agents increasingly automate survey writing, yet existing benchmarks do not jointly test whether they retrieve the papers experts consider essential and organize thos…
CL-bench Life: Can Language Models Learn from Real-Life Context?
Shihan Dou, Yujiong Shen, Chenhao Huang +35
Today's AI assistants such as OpenClaw are designed to handle context effectively, making context learning an increasingly important capability for models. As these systems move be…
LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models
Ming Zhang, Yujiong Shen, Jingyi Deng +19
Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting, critical issues that obscure true model ca…