Benchmarks as Microscopes: A Call for Model Metrology

Michael Saxon, Ari Holtzman, Peter West +2

Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-ba…