Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

Jeff Mohl, Nelson Gardner-Challis, Magda Dubois +6

Capabilities of frontier models are often assessed using agentic benchmarks. To trust these results, benchmarks must accurately measure what they claim to and be free from invalida…