Estimating Position Bias without Intrusive Interventions

Abstract

Presentation bias is one of the key challenges when learning from implicit feedback in search engines, as it confounds the relevance signal. While it was recently shown how counterfactual learning-to-rank (LTR) approaches \cite{Joachims/etal/17a} can provably overcome presentation bias when observation propensities are known, it remains to show how to effectively estimate these propensities. In this paper, we propose the first method for producing consistent propensity estimates without manual relevance judgments, disruptive interventions, or restrictive relevance modeling assumptions. First, we show how to harvest a specific type of intervention data from historic feedback logs of multiple different ranking functions, and show that this data is sufficient for consistent propensity estimation in the position-based model. Second, we propose a new extremum estimator that makes effective use of this data. In an empirical evaluation, we find that the new estimator provides superior propensity estimates in two real-world systems -- Arxiv Full-text Search and Google Drive Search. Beyond these two points, we find that the method is robust to a wide range of settings in simulation studies.

References in corpus (3)

• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Unbiased Learning to Rank with Unbiased Propensity Estimation
• Effective Evaluation using Logged Bandit Feedback from Multiple Loggers

Cited by in corpus (34)

• Causal Intervention for Leveraging Popularity Bias in Recommendation
• Controlling Fairness and Bias in Dynamic Learning-to-Rank
• FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platforms
• CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation
• Unifying Online and Counterfactual Learning to Rank
• Policy-Aware Unbiased Learning to Rank for Top-k Rankings
• Intervention Harvesting for Context-Dependent Examination-Bias Estimation
• When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank
• A Model-Agnostic Causal Learning Framework for Recommendation using Search Data
• Towards Fair Recommendation in Two-Sided Platforms
• Deconfounded Causal Collaborative Filtering
• Maximizing Marginal Fairness for Dynamic Learning to Rank
• Policy-Gradient Training of Fair and Unbiased Ranking Functions
• Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction
• Deep Position-wise Interaction Network for CTR Prediction
• Fairness of Exposure in Light of Incomplete Exposure Estimation
• Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
• Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking
• U-rank: Utility-oriented Learning to Rank with Implicit Feedback
• A Probabilistic Position Bias Model for Short-Video Recommendation Feeds
• Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
• Accelerated Convergence for Counterfactual Learning to Rank
• Implicit Feedback for Dense Passage Retrieval: A Counterfactual Approach
• The Importance of Causality in Decision Making: A Perspective on Recommender Systems
• Position Bias Estimation for Unbiased Learning-to-Rank in eCommerce Search
• De-Biased Modelling of Search Click Behavior with Reinforcement Learning
• Marginal-Certainty-aware Fair Ranking Algorithm
• Vertical Allocation-based Fair Exposure Amortizing in Ranking
• Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank
• On the Impact of Outlier Bias on User Clicks
• Click-Conversion Multi-Task Model with Position Bias Mitigation for Sponsored Search in eCommerce
• FARA: Future-aware Ranking Algorithm for Fairness Optimization
• Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank
• A General Framework for Pairwise Unbiased Learning to Rank