Implicit Feedback for Dense Passage Retrieval: A Counterfactual Approach
arXiv:2204.00718 · doi:10.1145/3477495.3531994
Abstract
In this paper we study how to effectively exploit implicit feedback in Dense Retrievers (DRs). We consider the specific case in which click data from a historic click log is available as implicit feedback. We then exploit such historic implicit interactions to improve the effectiveness of a DR. A key challenge that we study is the effect that biases in the click signal, such as position bias, have on the DRs. To overcome the problems associated with the presence of such bias, we propose the Counterfactual Rocchio (CoRocchio) algorithm for exploiting implicit feedback in Dense Retrievers. We demonstrate both theoretically and empirically that dense query representations learnt with CoRocchio are unbiased with respect to position bias and lead to higher retrieval effectiveness. We make available the implementations of the proposed methods and the experimental framework, along with all results at https://github.com/ielab/Counterfactual-DR.
arXiv commentFull paper, accepted at SGIR2022
References in corpus (15)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Overview of the TREC 2020 deep learning track
- Correcting for Selection Bias in Learning-to-rank Systems
- Estimating Position Bias without Intrusive Interventions
- Differentiable Unbiased Online Learning to Rank
- A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques
- To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions
- Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
- Intervention Harvesting for Context-Dependent Examination-Bias Estimation
- Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
- Effective Evaluation using Logged Bandit Feedback from Multiple Loggers
- Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations
- Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction
- Sensitive and Scalable Online Evaluation with Theoretical Guarantees