How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication Methods

David Otero, Javier Parapar, Nicola Ferro

Creating test collections for offline retrieval evaluation requires human effort to judge documents' relevance. This expensive activity motivated much work in developing methods fo…