Declarative Experimentation in Information Retrieval using PyTerrier

Abstract

The advent of deep machine learning platforms such as Tensorflow and Pytorch, developed in expressive high-level languages such as Python, have allowed more expressive representations of deep neural network architectures. We argue that such a powerful formalism is missing in information retrieval (IR), and propose a framework called PyTerrier that allows advanced retrieval pipelines to be expressed, and evaluated, in a declarative manner close to their conceptual design. Like the aforementioned frameworks that compile deep learning experiments into primitive GPU operations, our framework targets IR platforms as backends in order to execute and evaluate retrieval pipelines. Further, we can automatically optimise the retrieval pipelines to increase their efficiency to suite a particular IR platform backend. Our experiments, conducted on TREC Robust and ClueWeb09 test collections, demonstrate the efficiency benefits of these optimisations for retrieval pipelines involving both the Anserini and Terrier IR platforms.

References in corpus (4)

• XGBoost: A Scalable Tree Boosting System
• CEDR: Contextualized Embeddings for Document Ranking
• Introducing LETOR 4.0 Datasets
• Pytrec_eval: An Extremely Fast Python Interface to trec_eval

Cited by in corpus (20)

• Simplified Data Wrangling with ir_datasets
• Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval
• The Information Retrieval Experiment Platform
• Query Embedding Pruning for Dense Retrieval
• Towards Robust Expert Finding in Community Question Answering Platforms
• ABNIRML: Analyzing the Behavior of Neural IR Models
• ir_metadata: An Extensible Metadata Schema for IR Experiments
• On Approximate Nearest Neighbour Selection for Multi-Stage Dense Retrieval
• Retrieval Augmented Zero-Shot Text Classification
• Evaluation of Temporal Change in IR Test Collections
• Neural Passage Quality Estimation for Static Pruning
• Sparse Pairwise Re-ranking with Pre-trained Transformers
• Constructing and Evaluating Declarative RAG Pipelines in PyTerrier
• On Single and Multiple Representations in Dense Passage Retrieval
• Demographically-Inspired Query Variants Using an LLM
• On the Effects of Regional Spelling Conventions in Retrieval Models
• Validating Synthetic Usage Data in Living Lab Environments
• Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting
• Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders
• Towards a Relevance Posterior in Neural Information Access