Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank

Abstract

Click-based learning to rank (LTR) tackles the mismatch between click frequencies on items and their actual relevance. The approach of previous work has been to assume a model of click behavior and to subsequently introduce a method for unbiasedly estimating preferences under that assumed model. The success of this approach is evident in that unbiased methods have been found for an increasing number of behavior models and types of bias. This work aims to uncover the implicit limitations of the high-level prevalent approach in the counterfactual LTR field. Thus, in contrast with limitations that follow from explicit assumptions, our aim is to recognize limitations that the field is currently unaware of. We do this by inverting the existing approach: we start by capturing existing methods in generic terms, and subsequently, from these generic descriptions we derive the click behavior for which these methods can be unbiased. Our inverted approach reveals that there are indeed implicit limitations to the counterfactual LTR approach: we find counterfactual estimation can only produce unbiased methods for click behavior based on affine transformations. In addition, we also recognize previously undiscussed limitations of click-modelling and pairwise approaches to click-based LTR. Our findings reveal that it is impossible for existing approaches to provide unbiasedness guarantees for all plausible click behavior models.

In Proceedings of the 2022 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR '22), July 11-12, 2022, Madrid, Spain. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3539813.3545137 ISBN 978-1-4503-9412-3/22/07

References in corpus (7)

• Controlling Fairness and Bias in Dynamic Learning-to-Rank
• Estimating Position Bias without Intrusive Interventions
• Unifying Online and Counterfactual Learning to Rank
• To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions
• Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank
• Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking
• Mixture-Based Correction for Position and Trust Bias in Counterfactual Learning to Rank

Cited by in corpus (8)

• Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization
• Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
• The Role of Relevance in Fair Ranking
• An Offline Metric for the Debiasedness of Click Models
• Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank
• Recent Advances in the Foundations and Applications of Unbiased Learning to Rank
• Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study
• An Epistemic Position-Based Click Model: From Interactions to Epistemic Distributions of Relevance and Bias