5 papers
SuiteEval: Simplifying Retrieval Benchmarks
Andrew Parry, Debasis Ganguly, Sean MacAvaney
Information retrieval evaluation often suffers from fragmented practices -- varying dataset subsets, aggregation methods, and pipeline configurations -- that undermine reproducibil…
Disentangling Locality and Entropy in Ranking Distillation
Andrew Parry, Debasis Ganguly, Sean MacAvaney
The training process of ranking models involves two key data selection decisions: a sampling strategy, and a labeling strategy. Modern ranking systems, especially those for perform…
Modeling Ranking Properties with In-Context Learning
Nilanjan Sinhababu, Andrew Parry, Debasis Ganguly +1
While standard IR models are mainly designed to optimize relevance, real-world search often needs to balance additional objectives such as diversity and fairness. These objectives…
Variations in Relevance Judgments and the Shelf Life of Test Collections
Andrew Parry, Maik Fröbe, Harrisen Scells +5
The fundamental property of Cranfield-style evaluations, that system rankings are stable even when assessors disagree on individual relevance decisions, was validated on traditiona…
MechIR: A Mechanistic Interpretability Framework for Information Retrieval
Andrew Parry, Catherine Chen, Carsten Eickhoff +1
Mechanistic interpretability is an emerging diagnostic approach for neural models that has gained traction in broader natural language processing domains. This paradigm aims to pro…