Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

Weijun Luo, Kelvin Luu, Xinyi Liu +6

Agentic benchmarks guide model selection and training. Yet an agent can pass a task without demonstrating the intended capability. Such outcomes constitute unearned passes; their p…