Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls

Abstract

Pseudo Relevance Feedback (PRF) is known to improve the effectiveness of bag-of-words retrievers. At the same time, deep language models have been shown to outperform traditional bag-of-words rerankers. However, it is unclear how to integrate PRF directly with emergent deep language models. In this article, we address this gap by investigating methods for integrating PRF signals into rerankers and dense retrievers based on deep language models. We consider text-based and vector-based PRF approaches, and investigate different ways of combining and scoring relevance signals. An extensive empirical evaluation was conducted across four different datasets and two task settings (retrieval and ranking). Text-based PRF results show that the use of PRF had a mixed effect on deep rerankers across different datasets. We found that the best effectiveness was achieved when (i) directly concatenating each PRF passage with the query, searching with the new set of queries, and then aggregating the scores; (ii) using Borda to aggregate scores from PRF runs. Vector-based PRF results show that the use of PRF enhanced the effectiveness of deep rerankers and dense retrievers over several evaluation metrics. We found that higher effectiveness was achieved when (i) the query retains either the majority or the same weight within the PRF mechanism, and (ii) a shallower PRF signal (i.e., a smaller number of top-ranked passages) was employed, rather than a deeper signal. Our vector-based PRF method is computationally efficient; thus this represents a general PRF method others can use with deep rerankers and dense retrievers.

Accepted to the Journal of ACM Transactions on Information Systems (TOIS)

References in corpus (12)

• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Simple Applications of BERT for Ad Hoc Document Retrieval
• Overview of the TREC 2020 deep learning track
• Using Word Embeddings for Automatic Query Expansion
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• Complementing Lexical Retrieval with Semantic Residual Embedding
• RepBERT: Contextualized Text Embeddings for First-Stage Retrieval
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Overview of the TREC 2021 deep learning track
• TREC CAsT 2019: The Conversational Assistance Track Overview
• The Simplest Thing That Can Possibly Work: Pseudo-Relevance Feedback Using Text Classification