The Flaw of Averages: Measuring Benchmark-Level Distributional Robustness

Arda Uzunoglu, Tianjian Li, Daniel Khashabi

Benchmarks are central to measuring progress in language models, but aggregate scores can obscure substantial variation across subdomains, making models appear broadly competent de…