Pytrec_eval: An Extremely Fast Python Interface to trec_eval

Abstract

We introduce pytrec_eval, a Python interface to the tree_eval information retrieval evaluation toolkit. pytrec_eval exposes the reference implementations of trec_eval within Python as a native extension. We show that pytrec_eval is around one order of magnitude faster than invoking trec_eval as a sub process from within Python. Compared to a native Python implementation of NDCG, pytrec_eval is twice as fast for practically-sized rankings. Finally, we demonstrate its effectiveness in an application where pytrec_eval is combined with Pyndri and the OpenAI Gym where query expansion is learned using Q-learning.

arXiv commentSIGIR '18. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

Cited by in corpus (8)

• Declarative Experimentation in Information Retrieval using PyTerrier
• Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• Enhancing User Behavior Sequence Modeling by Generative Tasks for Session Search
• A Reproducibility Study of PLAID
• repro_eval: A Python Interface to Reproducibility Measures of System-oriented IR Experiments
• From Easy to Hard: A Dual Curriculum Learning Framework for Context-Aware Document Ranking
• The Impacts of Data, Ordering, and Intrinsic Dimensionality on Recall in Hierarchical Navigable Small Worlds