Can We Trust Item Response Theory for AI Evaluation?

Han Jiang, Sunbeom Kwon, Jinwen Luo +2

AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate capabilities, rank systems, select informative examples, and…