1 citations · 1 across the 11 of their papers we have counts for
12 papers
AcademiClaw: When Students Set Challenges for AI Agents
Junjie Yu, Pengrui Lu, Weiye Si +75
Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We intro…
AlphaEval: Evaluating Agents in Production
Pengrui Lu, Bingyu Xu, Wenjun Zhang +24
The rapid deployment of AI agents in commercial settings has outpaced the development of evaluation methodologies that reflect production realities. Existing benchmarks measure age…
NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions
Shizheng Hou, Wenqi Pei, Nuo Chen +3
Natural Language to SQL (NL2SQL) technology empowers non-expert users to query relational databases without requiring SQL expertise. While large language models (LLMs) have greatly…
Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
Tianze Xu, Yanzhao Zheng, Pengrui Lu +11
Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. Howeve…
daVinci-LLM:Towards the Science of Pretraining
Yiwei Qin, Yixiu Liu, Tiantian Mi +12
The foundational pretraining phase determines a model's capability ceiling, as post-training struggles to overcome capability foundations established during pretraining, yet it rem…
ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
Pengrui Lu, Shiqi Zhang, Yunzhong Hou +8
Recent coding agents can generate complete codebases from simple prompts, yet existing evaluations focus on issue-level bug fixing and lag behind end-to-end development. We introdu…
Interaction as Intelligence Part II: Asynchronous Human-Agent Rollout for Long-Horizon Task Training
Dayuan Fu, Yunze Wu, Xiaojie Cai +13
Large Language Model (LLM) agents have recently shown strong potential in domains such as automated coding, deep research, and graphical user interface manipulation. However, train…
InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research
Yunze Wu, Dayuan Fu, Weiye Si +13
AI agents could accelerate scientific discovery by automating hypothesis formation, experiment design, coding, execution, and analysis, yet existing benchmarks probe narrow skills…
ParaCook: On Time-Efficient Planning for Multi-Agent Systems
Shiqi Zhang, Xinbei Ma, Yunqing Xu +7
Large Language Models (LLMs) exhibit strong reasoning abilities for planning long-horizon, real-world tasks, yet existing agent benchmarks focus on task completion while neglecting…
ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry
Tianze Xu, Pengrui Lu, Lyumanshan Ye +2
The emergence of deep research systems presents significant capabilities in problem-solving, extending from basic queries to sophisticated research tasks. However, existing benchma…
Interaction as Intelligence: Deep Research With Human-AI Partnership
Lyumanshan Ye, Xiaojie Cai, Xinkai Wang +23
This paper introduces "Interaction as Intelligence" research series, presenting a reconceptualization of human-AI relationships in deep research tasks. Traditional approaches treat…
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu +4
Large Language Models (LLMs) equipped with web search capabilities have demonstrated impressive potential for deep research tasks. However, current approaches predominantly rely on…