How Should AI Safety Benchmarks Benchmark Safety?

Cheng Yu, Severin Engelmann, Ruoxuan Cao +2

AI safety benchmarks are pivotal for safety in advanced AI systems; however, they have significant technical, epistemic, and sociotechnical shortcomings. We present a review of 210…