Equal Ranking Quality, Different Decisions: Training Order-Consistent LLM Scorers

Markus Frohmann, Mahdiyar Alavi, Elizabeth Lingg +1

Rerankers, reward models and multi-document QA scorers score candidate documents or responses in one LLM prompt, so each score depends on their order. Such scorers are selected on…