Correcting for Selection Bias in Learning-to-rank Systems

Abstract

Click data collected by modern recommendation systems are an important source of observational data that can be utilized to train learning-to-rank (LTR) systems. However, these data suffer from a number of biases that can result in poor performance for LTR systems. Recent methods for bias correction in such systems mostly focus on position bias, the fact that higher ranked results (e.g., top search engine results) are more likely to be clicked even if they are not the most relevant results given a user's query. Less attention has been paid to correcting for selection bias, which occurs because clicked documents are reflective of what documents have been shown to the user in the first place. Here, we propose new counterfactual approaches which adapt Heckman's two-stage method and accounts for selection and position bias in LTR systems. Our empirical evaluation shows that our proposed methods are much more robust to noise and have better accuracy compared to existing unbiased LTR algorithms, especially when there is moderate to no position bias.

This paper appeared in The Web Conference (WWW'20), April 20-24, 2020, Taipei, Taiwan

References in corpus (6)

• Biased Assimilation, Homophily and the Dynamics of Polarization
• How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility
• Unbiased Learning to Rank with Unbiased Propensity Estimation
• Differentiable Unbiased Online Learning to Rank
• To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions
• The Deconfounded Recommender: A Causal Inference Approach to Recommendation

Cited by in corpus (31)

• Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue
• AutoDebias: Learning to Debias for Recommendation
• SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Recommendation in Hundreds of Travel Scenarios
• Unifying Online and Counterfactual Learning to Rank
• A Model-Agnostic Causal Learning Framework for Recommendation using Search Data
• Deconfounded Causal Collaborative Filtering
• Maximizing Marginal Fairness for Dynamic Learning to Rank
• CausalRec: Causal Inference for Visual Debiasing in Visually-Aware Recommendation
• It Is Different When Items Are Older: Debiasing Recommendations When Selection Bias and User Preferences Are Dynamic
• Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank
• Dos and Don'ts of Machine Learning in Computer Security
• Deep Position-wise Interaction Network for CTR Prediction
• Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking
• Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
• Understanding and Mitigating the Effect of Outliers in Fair Ranking
• RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems
• Going Beyond Popularity and Positivity Bias: Correcting for Multifactorial Bias in Recommender Systems
• A Probabilistic Position Bias Model for Short-Video Recommendation Feeds
• Popularity Bias Is Not Always Evil: Disentangling Benign and Harmful Bias for Recommendation
• TFROM: A Two-sided Fairness-Aware Recommendation Model for Both Customers and Providers
• Implicit Feedback for Dense Passage Retrieval: A Counterfactual Approach
• The Importance of Causality in Decision Making: A Perspective on Recommender Systems
• An Offline Metric for the Debiasedness of Click Models
• On the Impact of Outlier Bias on User Clicks
• Practical and Robust Safety Guarantees for Advanced Counterfactual Learning to Rank
• Recent Advances in the Foundations and Applications of Unbiased Learning to Rank
• Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank
• An Adaptive Boosting Technique to Mitigate Popularity Bias in Recommender System
• Users' Perception of Search Engine Biases and Satisfaction
• Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study
• An Epistemic Position-Based Click Model: From Interactions to Epistemic Distributions of Relevance and Bias