Training on the Test Task Confounds Evaluation and Emergence

Ricardo Dominguez-Olmedo, Florian E. Dorner, Moritz Hardt

We study a fundamental problem in the evaluation of large language models that we call training on the test task. Unlike wrongful practices like training on the test data, leakage,…