When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank

Abstract

Besides position bias, which has been well-studied, trust bias is another type of bias prevalent in user interactions with rankings: users are more likely to click incorrectly w.r.t. their preferences on highly ranked items because they trust the ranking system. While previous work has observed this behavior in users, we prove that existing Counterfactual Learning to Rank (CLTR) methods do not remove this bias, including methods specifically designed to mitigate this type of bias. Moreover, we prove that Inverse Propensity Scoring (IPS) is principally unable to correct for trust bias under non-trivial circumstances. Our main contribution is a new estimator based on affine corrections: it both reweights clicks and penalizes items displayed on ranks with high trust bias. Our estimator is the first estimator that is proven to remove the effect of both trust bias and position bias. Furthermore, we show that our estimator is a generalization of the existing CLTR framework: if no trust bias is present, it reduces to the original IPS estimator. Our semi-synthetic experiments indicate that by removing the effect of trust bias in addition to position bias, CLTR can approximate the optimal ranking system even closer than previously possible.

arXiv commentCIKM 2020

References in corpus (4)

• Estimating Position Bias without Intrusive Interventions
• To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions
• Policy-Aware Unbiased Learning to Rank for Top-k Rankings
• Intervention Harvesting for Context-Dependent Examination-Bias Estimation

Cited by in corpus (17)

• Fairness of Exposure in Light of Incomplete Exposure Estimation
• Understanding and Mitigating the Effect of Outliers in Fair Ranking
• Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization
• A Probabilistic Position Bias Model for Short-Video Recommendation Feeds
• Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
• Mixture-Based Correction for Position and Trust Bias in Counterfactual Learning to Rank
• Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking
• Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank
• An Offline Metric for the Debiasedness of Click Models
• On the Impact of Outlier Bias on User Clicks
• Recent Advances in the Foundations and Applications of Unbiased Learning to Rank
• Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank
• Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank
• Rethinking Click Models in Light of Carousel Interfaces: Theory-Based Categorization and Design of Click Models
• Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rank
• Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study
• An Epistemic Position-Based Click Model: From Interactions to Epistemic Distributions of Relevance and Bias