Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank

Abstract

Unbiased CLTR requires click propensities to compensate for the difference between user clicks and true relevance of search results via IPS. Current propensity estimation methods assume that user click behavior follows the PBM and estimate click propensities based on this assumption. However, in reality, user clicks often follow the CM, where users scan search results from top to bottom and where each next click depends on the previous one. In this cascade scenario, PBM-based estimates of propensities are not accurate, which, in turn, hurts CLTR performance. In this paper, we propose a propensity estimation method for the cascade scenario, called CM-IPS. We show that CM-IPS keeps CLTR performance close to the full-information performance in case the user clicks follow the CM, while PBM-based CLTR has a significant gap towards the full-information. The opposite is true if the user clicks follow PBM instead of the CM. Finally, we suggest a way to select between CM- and PBM-based propensity estimation methods based on historical user clicks.

4 pages, 2 figures, 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '20)

References in corpus (1)

• Differentiable Unbiased Online Learning to Rank

Cited by in corpus (16)

• Deconfounded Causal Collaborative Filtering
• Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank
• Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
• Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
• Mixture-Based Correction for Position and Trust Bias in Counterfactual Learning to Rank
• The Bandwagon Effect: Not Just Another Bias
• The Importance of Causality in Decision Making: A Perspective on Recommender Systems
• The Role of Relevance in Fair Ranking
• An Offline Metric for the Debiasedness of Click Models
• FairSort: Learning to Fair Rank for Personalized Recommendations in Two-Sided Platforms
• AIE: Auction Information Enhanced Framework for CTR Prediction in Online Advertising
• Click-Conversion Multi-Task Model with Position Bias Mitigation for Sponsored Search in eCommerce
• Recent Advances in the Foundations and Applications of Unbiased Learning to Rank
• Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank
• Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study
• An Epistemic Position-Based Click Model: From Interactions to Epistemic Distributions of Relevance and Bias