13 papers
Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows
Tianyang Liu, Canwen Xu, Fangyu Lei +6
Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extract…
Auto-Configuring Scientific Simulators with Lightweight Coding-Agent Adapters
Matthew Ho, Brian Liu, Jixuan Chen +2
Configuring an advanced scientific simulator, translating a modeling goal into a valid, runnable input deck, is a persistent bottleneck that costs domain scientists hours to days.…
AgentSpec: Understanding Embodied Agent Scaffolds Through Controlled Composition
Jixuan Chen, Jianzhi Shen, Haoqiang Kang +10
LLM agents are increasingly built not as single model calls, but as scaffolded systems that combine reasoning, memory, reflection, action execution, and learning. While such scaffo…
OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces
Xiaozhe Li, Jixuan Chen, Xinyu Fang +4
Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning and tool use. However, the fundamental cognitive faculties essential for problem solving, includ…
CocoaBench: Evaluating Unified Digital Agents in the Wild
CocoaBench Team, Shibo Hao, Zhining Zhang +29
LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly int…
DeliveryBench: Can Agents Earn Profit in Real World?
Lingjun Mao, Jiawei Ren, Kun Zhou +3
LLMs and VLMs are increasingly deployed as embodied agents, yet existing benchmarks largely revolve around simple short-term tasks and struggle to capture rich realistic constraint…