44 citations · 55 across the 10 of their papers we have counts for
11 papers · 1 filter
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Soham Dan, Jeff Da, Sanyam Bhutani +8
SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and in…
From Copying to Composition: How Language Models Develop In-Context Learning
Minglai Yang, Xinyu Guo, Renxiong Wang +3
In-context learning (ICL) allows language models to adapt their behavior from instructions and demonstrations in a prompt, but accuracy alone does not reveal how contextual informa…
Evidence-First Agentic Verification
Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou +7
Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the…
Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
Weijun Luo, Kelvin Luu, Xinyi Liu +6
Agentic benchmarks guide model selection and training. Yet an agent can pass a task without demonstrating the intended capability. Such outcomes constitute unearned passes; their p…
READY or Not: Reliable Enterprise Agent Deployment
Veronica Chatrath, Bryan Zhu, Jingxuan Fan +15
An AI agent can perform well on benchmarks and still be unsuitable for deployment. Existing AI-agent benchmarks measure whether an agent can complete realistic professional work, w…
Spine-Branch Coordination for Multi-agent Computer Use
Mian Zhang, Manasi Sharma, Sheng Zhang +5
Computer use agents (CUAs) are increasingly deployed as multi-agent systems that decompose a task into multiple subtasks executed across parallel virtual machines (VMs). However, a…
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL
Minglai Yang, Xinyu Guo, Zhengliang Shi +8
Rubric-based reinforcement learning, in which an LLM judge scores responses against prompt-specific criteria, has become a standard way to post-train language models on tasks witho…
CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
Veronica Chatrath, Bryan Zhu, George Pu +16
Large language models perform strongly on medical knowledge benchmarks, but reliable clinical deployment requires agents to conduct defensible investigations over heterogeneous, lo…
ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks
Vincent Siu, Manasi Sharma, Dawn Song +3
Computer use agents are evaluated almost exclusively on atomic desktop tasks, but realistic desktop work requires sustaining state across multiple objectives. We study this gap wit…
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
Chaithanya Bandi, Razvan-Gabriel Dumitru, Ben Hertzberg +20
The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evalu…
The Limits of AI Data Transparency Policy: Three Disclosure Fallacies
Judy Hanwen Shen, Ken Liu, Angelina Wang +7
Data transparency has emerged as a rallying cry for addressing concerns about AI: data quality, privacy, and copyright chief among them. Yet while these calls are crucial for accou…