Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

Markus Frohmann, Mahdiyar Alavi, Mahdiyar Ali Akbar Alavi +2

In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one prompt, each still receiving i…