Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions

Abstract

Users of music streaming, video streaming, news recommendation, and e-commerce services often engage with content in a sequential manner. Providing and evaluating good sequences of recommendations is therefore a central problem for these services. Prior reweighting-based counterfactual evaluation methods either suffer from high variance or make strong independence assumptions about rewards. We propose a new counterfactual estimator that allows for sequential interactions in the rewards with lower variance in an asymptotically unbiased manner. Our method uses graphical assumptions about the causal relationships of the slate to reweight the rewards in the logging policy in a way that approximates the expected sum of rewards under the target policy. Extensive experiments in simulation and on a live recommender system show that our approach outperforms existing methods in terms of bias and data efficiency for the sequential track recommendations problem.

References in corpus (7)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Evaluation of Session-based Recommendation Algorithms
• Recommendations as Treatments: Debiasing Learning and Evaluation
• Offline A/B testing for Recommender Systems
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Off-policy evaluation for slate recommendation
• More Efficient Off-Policy Evaluation through Regularized Targeted Learning

Cited by in corpus (13)

• Advances and Challenges in Conversational Recommender Systems: A Survey
• Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model
• Generative Slate Recommendation with Reinforcement Learning
• Variation Control and Evaluation for Generative SlateRecommendations
• Why People Skip Music? On Predicting Music Skips using Deep Reinforcement Learning
• Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstraction
• Off-Policy Evaluation of Ranking Policies under Diverse User Behavior
• Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
• Improving Location Recommendation with Urban Knowledge Graph
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Off-Policy Evaluation of Slate Policies under Bayes Risk
• Control Variates for Slate Off-Policy Evaluation
• Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems