An Offline Metric for the Debiasedness of Click Models

Abstract

A well-known problem when learning from user clicks are inherent biases prevalent in the data, such as position or trust bias. Click models are a common method for extracting information from user clicks, such as document relevance in web search, or to estimate click biases for downstream applications such as counterfactual learning-to-rank, ad placement, or fair ranking. Recent work shows that the current evaluation practices in the community fail to guarantee that a well-performing click model generalizes well to downstream tasks in which the ranking distribution differs from the training distribution, i.e., under covariate shift. In this work, we propose an evaluation metric based on conditional independence testing to detect a lack of robustness to covariate shift in click models. We introduce the concept of debiasedness in click modeling and derive a metric for measuring it. In extensive semi-synthetic experiments, we show that our proposed metric helps to predict the downstream performance of click models under covariate shift and is useful in an off-policy model selection setting.

SIGIR23 - Full paper

References in corpus (6)

• Towards Out-Of-Distribution Generalization: A Survey
• When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank
• Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank
• An Adversarial Imitation Click Model for Information Retrieval
• Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking
• Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank

Cited by in corpus (2)

• Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
• GraphEx: A Graph-based Extraction Method for Advertiser Keyphrase Recommendation