Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations

Abstract

Pyserini is an easy-to-use Python toolkit that supports replicable IR research by providing effective first-stage retrieval in a multi-stage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, pre-built indexes, and evaluation scripts for many commonly used IR test collections. We aim to support, out of the box, the entire research lifecycle of efforts aimed at improving ranking with modern neural approaches. In particular, Pyserini supports sparse retrieval (e.g., BM25 scoring using bag-of-words representations), dense retrieval (e.g., nearest-neighbor search on transformer-encoded representations), as well as hybrid retrieval that integrates both approaches. This paper provides an overview of toolkit features and presents empirical results that illustrate its effectiveness on two popular ranking tasks. We also describe how our group has built a culture of replicability through shared norms and tools that enable rigorous automated testing.

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Document Expansion by Query Prediction
• Multi-Stage Document Ranking with BERT
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
• CO-Search: COVID-19 Information Retrieval with Semantic Search, Question Answering, and Abstractive Summarization
• RRF102: Meeting the TREC-COVID Challenge with a 100+ Runs Ensemble

Cited by in corpus (12)

• Retrieving Supporting Evidence for Generative Question Answering
• A Replication Study of Dense Passage Retriever
• To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment
• Implicit Feedback for Dense Passage Retrieval: A Counterfactual Approach
• One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval
• Overview of LiLAS 2021 -- Living Labs for Academic Search
• OpenMatch: An Open Source Library for Neu-IR Research
• Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answering
• Contextualized Query Embeddings for Conversational Search
• Simple Entity-Centric Questions Challenge Dense Retrievers
• Yes, BM25 is a Strong Baseline for Legal Case Retrieval
• Towards Universal Dense Retrieval for Open-domain Question Answering