Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling

Mohamed Eltahir, Abobaker Ahmed, Nawaf Barebood +3

Multiple-choice benchmarks are cheap to grade and are running out of room, and the standard remedy, writing harder items, is slow and repeated for every benchmark. A saturated benc…