Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
arXiv:1003.5956 · doi:10.1145/1935826.1935878
Abstract
Contextual bandit algorithms have become popular for online recommendation systems such as Digg, Yahoo! Buzz, and news recommendation in general. \emph{Offline} evaluation of the effectiveness of new algorithms in these applications is critical for protecting online user experiences but very challenging due to their "partial-label" nature. Common practice is to create a simulator which simulates the online environment for the problem at hand and then run an algorithm against this simulator. However, creating simulator itself is often difficult and modeling bias is usually unavoidably introduced. In this paper, we introduce a \emph{replay} methodology for contextual bandit algorithm evaluation. Different from simulator-based approaches, our method is completely data-driven and very easy to adapt to different applications. More importantly, our method can provide provably unbiased evaluations. Our empirical results on a large-scale news article recommendation dataset collected from Yahoo! Front Page conform well with our theoretical results. Furthermore, comparisons between our offline replay and online bucket evaluation of several contextual bandit algorithms show accuracy and effectiveness of our offline evaluation method.
10 pages, 7 figures, revised from the published version at the WSDM 2011 conference
References in corpus (3)
Cited by in corpus (122)
- How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility
- Behavior Regularized Offline Reinforcement Learning
- Contextual Bandits with Similarity Information
- Measuring the Business Value of Recommender Systems
- MOReL : Model-Based Offline Reinforcement Learning
- Empirical Analysis of Session-Based Recommendation Algorithms
- Recommendations as Treatments: Debiasing Learning and Evaluation
- Offline A/B testing for Recommender Systems
- Doubly Robust Policy Evaluation and Optimization
- Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
- DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
- Learning from Logged Implicit Exploration Data
- Estimating Position Bias without Intrusive Interventions
- Online Clustering of Bandits
- Computation Offloading in Heterogeneous Vehicular Edge Networks: On-line and Off-policy Bandit Solutions
- Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
- Making Contextual Decisions with Low Technical Debt
- Learning Contextual Bandits in a Non-stationary Environment
- Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
- To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions
- Off-policy evaluation for slate recommendation
- Thompson Sampling in Switching Environments with Bayesian Online Change Point Detection
- Carousel Personalization in Music Streaming Apps with Contextual Bandits
- Statistical Arbitrage Mining for Display Advertising
- Effective Evaluation using Logged Bandit Feedback from Multiple Loggers
- Confounding-Robust Policy Improvement
- Latent Contextual Bandits and their Application to Personalized Recommendations for New Users
- Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach
- Off-policy Bandits with Deficient Support
- Identifying Best Interventions through Online Importance Sampling
- Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL
- CoinDICE: Off-Policy Confidence Interval Estimation
- Optimal and Adaptive Off-policy Evaluation in Contextual Bandits
- Counterfactual Estimation and Optimization of Click Metrics for Search Engines
- Self-Supervised Reinforcement Learning for Recommender Systems
- Evaluating stochastic seeding strategies in networks
- A Change-Detection based Framework for Piecewise-stationary Multi-Armed Bandit Problem
- Doubly robust off-policy evaluation with shrinkage
- Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning
- Improving offline evaluation of contextual bandit algorithms via bootstrapping techniques
- Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
- On Minimax Optimal Offline Policy Evaluation
- Post-Contextual-Bandit Inference
- Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
- Exploration in Online Advertising Systems with Deep Uncertainty-Aware Learning
- PG-TS: Improved Thompson Sampling for Logistic Contextual Bandits
- Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
- Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
- Off-Policy Evaluation of Ranking Policies under Diverse User Behavior
- Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning
- contextual: Evaluating Contextual Multi-Armed Bandit Problems in R
- Accountable Off-Policy Evaluation With Kernel Bellman Statistics
- Empirical Bayes Regret Minimization
- Bandit Algorithms for Precision Medicine
- Nearly Optimal Algorithms for Piecewise-Stationary Cascading Bandits
- Sequential Monte Carlo Bandits
- Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation
- Recent Advances in the Foundations and Applications of Unbiased Learning to Rank
- Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
- Conversational Contextual Bandit: Algorithm and Application
- Accurate Inference for Adaptive Linear Models
- Uplift Modeling for Multiple Treatments with Cost Optimization
- Nearly Dimension-Independent Sparse Linear Bandit over Small Action Spaces via Best Subset Selection
- A Near-Optimal Change-Detection Based Algorithm for Piecewise-Stationary Combinatorial Semi-Bandits
- A Linear Bandit for Seasonal Environments
- A Practical Guide of Off-Policy Evaluation for Bandit Problems
- Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits
- Reducing Exploration of Dying Arms in Mortal Bandits
- Adaptively Optimize Content Recommendation Using Multi Armed Bandit Algorithms in E-commerce
- User Clustering in Online Advertising via Topic Models
- Evaluation of Explore-Exploit Policies in Multi-result Ranking Systems
- Finite-time Analysis of Globally Nonstationary Multi-Armed Bandits
- The Role of Contextual Information in Best Arm Identification
- Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning
- Efficient Online Bayesian Inference for Neural Bandits
- Personalized Product Assortment with Real-time 3D Perception and Bayesian Payoff Estimation
- Offline Comparison of Ranking Functions using Randomized Data
- Deep Bayesian Bandits: Exploring in Online Personalized Recommendations
- Computational Causal Inference
- Personalized Advertisement Recommendation: A Ranking Approach to Address the Ubiquitous Click Sparsity Problem
- Impression-Aware Recommender Systems
- Offline Evaluation of Ranking Policies with Click Models
- A General Framework for Counterfactual Learning-to-Rank
- Debiasing Samples from Online Learning Using Bootstrap
- Decision Making Problems with Funnel Structure: A Multi-Task Learning Approach with Application to Email Marketing Campaigns
- Boosting Offline Reinforcement Learning with Residual Generative Modeling
- TS-UCB: Improving on Thompson Sampling With Little to No Additional Computation
- Concentration bounds for temporal difference learning with linear function approximation: The case of batch data and uniform sampling
- Action Centered Contextual Bandits
- Bandit Learning for Diversified Interactive Recommendation
- AdaLinUCB: Opportunistic Learning for Contextual Bandits
- Contextual-Bandit Based Personalized Recommendation with Time-Varying User Interests
- Generalizing Off-Policy Learning under Sample Selection Bias
- Multi-Armed Bandits on Partially Revealed Unit Interval Graphs
- Control Variates for Slate Off-Policy Evaluation
- Non-Stationary Off-Policy Optimization
- Contextual Recommendations and Low-Regret Cutting-Plane Algorithms
- Bandit algorithms for real-time data capture on large social medias
- On Limited-Memory Subsampling Strategies for Bandits
- Self-Supervised Contextual Bandits in Computer Vision
- When and Whom to Collaborate with in a Changing Environment: A Collaborative Dynamic Bandit Solution
- Optimized Recommender Systems with Deep Reinforcement Learning
- Off-policy Learning for Multiple Loggers
- Optimal Mixture Weights for Off-Policy Evaluation with Multiple Behavior Policies
- An Empirical Analysis on Transparent Algorithmic Exploration in Recommender Systems
- A Map of Bandits for E-commerce
- AutoOffAB: Toward Automated Offline A/B Testing for Data-Driven Requirement Engineering
- Contributions to Representation Learning with Graph Autoencoders and Applications to Music Recommendation
- Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Bandits
- Active Offline Policy Selection
- Random Effect Bandits
- Variance-Aware Off-Policy Evaluation with Linear Function Approximation
- The Impact of Batch Learning in Stochastic Bandits
- The Adaptive Doubly Robust Estimator for Policy Evaluation in Adaptive Experiments and a Paradox Concerning Logging Policy
- Off-Policy Interval Estimation with Lipschitz Value Iteration
- DTR Bandit: Learning to Make Response-Adaptive Decisions With Low Regret
- Multi-armed Bandits with Cost Subsidy
- Generalized Translation and Scale Invariant Online Algorithm for Adversarial Multi-Armed Bandits
- An Opportunistic Bandit Approach for User Interface Experimentation
- Better Boosting with Bandits for Online Learning
- Pyramid: Enhancing Selectivity in Big Data Protection with Count Featurization
- Reducing offline evaluation bias of collaborative filtering algorithms