Confidence and Stability of Global and Pairwise Scores in NLP Evaluation

Georgii Levtsov, Dmitry Ustalov

With the advent of highly capable instruction-tuned neural language models, benchmarking in natural language processing (NLP) is increasingly shifting towards pairwise comparison l…