Counterfactual Risk Minimization: Learning from Logged Bandit Feedback

Abstract

We develop a learning principle and an efficient algorithm for batch learning from logged bandit feedback. This learning setting is ubiquitous in online systems (e.g., ad placement, web search, recommendation), where an algorithm makes a prediction (e.g., ad ranking) for a given input (e.g., query) and observes bandit feedback (e.g., user clicks on presented ads). We first address the counterfactual nature of the learning problem through propensity scoring. Next, we prove generalization error bounds that account for the variance of the propensity-weighted empirical risk estimator. These constructive bounds give rise to the Counterfactual Risk Minimization (CRM) principle. We show how CRM can be used to derive a new learning method -- called Policy Optimizer for Exponential Models (POEM) -- for learning stochastic linear rules for structured output prediction. We present a decomposition of the POEM objective that enables efficient stochastic gradient optimization. POEM is evaluated on several multi-label classification problems showing substantially improved robustness and generalization performance compared to the state-of-the-art.

10 pages

References in corpus (4)

• Empirical Bernstein Bounds and Sample Variance Penalization
• Learning from Logged Implicit Exploration Data
• Counterfactual Estimation and Optimization of Click Metrics for Search Engines
• Improving offline evaluation of contextual bandit algorithms via bootstrapping techniques

Cited by in corpus (25)

• KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos
• To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions
• A Survey on Causal Inference
• Benchmarks for Deep Off-Policy Evaluation
• Model Inversion Networks for Model-Based Optimization
• Understanding the role of importance weighting for deep learning
• Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains
• Efficient Policy Learning from Surrogate-Loss Classification Reductions
• Imitation-Regularized Offline Learning
• Improving Evolutionary Strategies with Generative Neural Networks
• Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theory
• Counterfactual Evaluation of Treatment Assignment Functions with Networked Observational Data
• Learning Continuous Treatment Policy and Bipartite Embeddings for Matching with Heterogeneous Causal Effects
• Productization Challenges of Contextual Multi-Armed Bandits
• Semi-Parametric Efficient Policy Learning with Continuous Actions
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• Beyond traditional assumptions in fair machine learning
• A framework for massive scale personalized promotion
• Control Variates for Slate Off-Policy Evaluation
• A General Framework for Pairwise Unbiased Learning to Rank
• Improved Algorithms for Conservative Exploration in Bandits
• An Empirical Analysis on Transparent Algorithmic Exploration in Recommender Systems
• Learning Robust Decision Policies from Observational Data
• Learning Pareto-Efficient Decisions with Confidence
• Counterfactual Adversarial Learning with Representation Interpolation