Intervention Harvesting for Context-Dependent Examination-Bias Estimation

Abstract

Accurate estimates of examination bias are crucial for unbiased learning-to-rank from implicit feedback in search engines and recommender systems, since they enable the use of Inverse Propensity Score (IPS) weighting techniques to address selection biases and missing data. Unfortunately, existing examination-bias estimators are limited to the Position-Based Model (PBM), where the examination bias may only depend on the rank of the document. To overcome this limitation, we propose a Contextual Position-Based Model (CPBM) where the examination bias may also depend on a context vector describing the query and the user. Furthermore, we propose an effective estimator for the CPBM based on intervention harvesting. A key feature of the estimator is that it does not require disruptive interventions but merely exploits natural variation resulting from the use of multiple historic ranking functions. Real-world experiments on the ArXiv search engine and semi-synthetic experiments on the Yahoo Learning-To-Rank dataset demonstrate the superior effectiveness and robustness of the new approach.

References in corpus (4)

• Doubly Robust Policy Evaluation and Learning
• Unbiased Learning to Rank with Unbiased Propensity Estimation
• Estimating Position Bias without Intrusive Interventions
• Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling

Cited by in corpus (23)

• Controlling Fairness and Bias in Dynamic Learning-to-Rank
• Unifying Online and Counterfactual Learning to Rank
• When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank
• Policy Learning for Fairness in Ranking
• Policy-Gradient Training of Fair and Unbiased Ranking Functions
• Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank
• Fairness of Exposure in Light of Incomplete Exposure Estimation
• Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking
• Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
• U-rank: Utility-oriented Learning to Rank with Implicit Feedback
• A Probabilistic Position Bias Model for Short-Video Recommendation Feeds
• Accelerated Convergence for Counterfactual Learning to Rank
• Implicit Feedback for Dense Passage Retrieval: A Counterfactual Approach
• On the Impact of Outlier Bias on User Clicks
• CAB: Continuous Adaptive Blending Estimator for Policy Evaluation and Learning
• Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank
• A General Framework for Counterfactual Learning-to-Rank
• Learning to Rank with Variable Result Presentation Lengths
• A General Framework for Pairwise Unbiased Learning to Rank
• Calibrating Explore-Exploit Trade-off for Fair Online Learning to Rank
• A Deep Recurrent Survival Model for Unbiased Ranking
• Counterfactual Learning to Rank using Heterogeneous Treatment Effect Estimation
• Ranker-agnostic Contextual Position Bias Estimation