Offline A/B testing for Recommender Systems

Abstract

Before A/B testing online a new version of a recommender system, it is usual to perform some offline evaluations on historical data. We focus on evaluation methods that compute an estimator of the potential uplift in revenue that could generate this new technology. It helps to iterate faster and to avoid losing money by detecting poor policies. These estimators are known as counterfactual or off-policy estimators. We show that traditional counterfactual estimators such as capped importance sampling and normalised importance sampling are experimentally not having satisfying bias-variance compromises in the context of personalised product recommendation for online advertising. We propose two variants of counterfactual estimates with different modelling of the bias that prove to be accurate in real-world conditions. We provide a benchmark of these estimators by showing their correlation with business metrics observed by running online A/B tests on a commercial recommender system.

References in corpus (1)

• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms

Cited by in corpus (29)

• Measuring the Business Value of Recommender Systems
• KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems
• KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos
• Neural Interactive Collaborative Filtering
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
• Towards Neural Mixture Recommender for Long Range Dependent User Sequences
• Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions
• Balancing Unobserved Confounding with a Few Unbiased Ratings in Debiased Recommendations
• Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model
• Rethinking Cross-Domain Sequential Recommendation under Open-World Assumptions
• A Survey on Point-of-Interest Recommendations Leveraging Heterogeneous Data
• DVR: Micro-Video Recommendation Optimizing Watch-Time-Gain under Duration Bias
• Evaluating the Robustness of Off-Policy Evaluation
• Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce
• A Probabilistic Position Bias Model for Short-Video Recommendation Feeds
• The Importance of Causality in Decision Making: A Perspective on Recommender Systems
• Estimating Error and Bias in Offline Evaluation Results
• Adversarial learning for product recommendation
• Off-Policy Evaluation of Ranking Policies under Diverse User Behavior
• Off-Policy Evaluation of Probabilistic Identity Data in Lookalike Modeling
• Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation
• Towards the D-Optimal Online Experiment Design for Recommender Selection
• Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking
• Exploitation Over Exploration: Unmasking the Bias in Linear Bandit Recommender Offline Evaluation
• On the Reliability of Sampling Strategies in Offline Recommender Evaluation
• Causal Meta-Mediation Analysis: Inferring Dose-Response Function From Summary Statistics of Many Randomized Experiments
• Meta Off-Policy Estimation
• AutoOffAB: Toward Automated Offline A/B Testing for Data-Driven Requirement Engineering