Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting
arXiv:2509.17440 · doi:10.1007/978-3-032-04354-2_8
Abstract
The longitudinal evaluation of retrieval systems aims to capture how information needs and documents evolve over time. However, classical Cranfield-style retrieval evaluations only consist of a static set of queries and documents and thereby miss time as an evaluation dimension. Therefore, longitudinal evaluations need to complement retrieval toolkits with custom logic. This custom logic increases the complexity of research software, which might reduce the reproducibility and extensibility of experiments. Based on our submissions to the 2024 edition of LongEval, we propose a custom extension of ir_datasets for longitudinal retrieval experiments. This extension allows for declaratively, instead of imperatively, describing important aspects of longitudinal retrieval experiments, e.g., which queries, documents, and/or relevance feedback are available at which point in time. We reimplement our submissions to LongEval 2024 against our new ir_datasets extension, and find that the declarative access can reduce the complexity of the code.
Best of labs paper for LongEval at CLEF 2024
References in corpus (8)
- Declarative Experimentation in Information Retrieval using PyTerrier
- Simplified Data Wrangling with ir_datasets
- The Information Retrieval Experiment Platform
- TripClick: The Log Files of a Large Health Web Search Engine
- How to Measure the Reproducibility of System-oriented IR Experiments
- repro_eval: A Python Interface to Reproducibility Measures of System-oriented IR Experiments
- Evaluation of Temporal Change in IR Test Collections
- Reproducible Hybrid Time-Travel Retrieval in Evolving Corpora