Simplified Data Wrangling with ir_datasets

Abstract

Managing the data for Information Retrieval (IR) experiments can be challenging. Dataset documentation is scattered across the Internet and once one obtains a copy of the data, there are numerous different data formats to work with. Even basic formats can have subtle dataset-specific nuances that need to be considered for proper use. To help mitigate these challenges, we introduce a new robust and lightweight tool (ir_datasets) for acquiring, managing, and performing typical operations over datasets used in IR. We primarily focus on textual datasets used for ad-hoc search. This tool provides both a Python and command line interface to numerous IR datasets and benchmarks. To our knowledge, this is the most extensive tool of its kind. Integrations with popular IR indexing and experimentation toolkits demonstrate the tool's utility. We also provide documentation of these datasets through the ir_datasets catalog: https://ir-datasets.com/. The catalog acts as a hub for information on datasets used in IR, providing core information about what data each benchmark provides as well as links to more detailed information. We welcome community contributions and intend to continue to maintain and grow this tool.

SIGIR 2021 Resource

References in corpus (4)

• Declarative Experimentation in Information Retrieval using PyTerrier
• Efficient Document Re-Ranking for Transformers by Precomputing Term Representations
• TripClick: The Log Files of a Large Health Web Search Engine
• Content-Based Weak Supervision for Ad-Hoc Re-Ranking

Cited by in corpus (22)

• The Information Retrieval Experiment Platform
• Towards Robust Expert Finding in Community Question Answering Platforms
• C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval
• ABNIRML: Analyzing the Behavior of Neural IR Models
• ir_metadata: An Extensible Metadata Schema for IR Experiments
• GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation
• An Investigation of Prompt Variations for Zero-shot LLM-based Rankers
• On Survivorship Bias in MS MARCO
• Neural Passage Quality Estimation for Static Pruning
• Data Fusion of Synthetic Query Variants With Generative Large Language Models
• Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking
• TARexp: A Python Framework for Technology-Assisted Review Experiments
• Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders
• Investigating the Effects of Sparse Attention on Cross-Encoders
• Demographically-Inspired Query Variants Using an LLM
• LongEval at CLEF 2025: Longitudinal Evaluation of IR Systems on Web and Scientific Data
• Reproducible Hybrid Time-Travel Retrieval in Evolving Corpora
• Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders
• Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting
• Simple Domain Adaptation for Sparse Retrievers
• On the Reproducibility of Learned Sparse Retrieval Adaptations for Long Documents
• The Matryoshka Hypencoder