Learning from Logged Implicit Exploration Data

Abstract

We provide a sound and consistent foundation for the use of \emph{nonrandom} exploration data in "contextual bandit" or "partially labeled" settings where only the value of a chosen action is learned. The primary challenge in a variety of settings is that the exploration policy, in which "offline" data is logged, is not explicitly known. Prior solutions here require either control of the actions during the learning process, recorded random exploration, or actions chosen obliviously in a repeated manner. The techniques reported here lift these restrictions, allowing the learning of a policy for choosing actions given features from historical data where no randomization occurred or was logged. We empirically verify our solution on two reasonably sized sets of real-world data obtained from Yahoo!.

References in corpus (2)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms

Cited by in corpus (41)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Learning Representations for Counterfactual Inference
• Recommendations as Treatments: Debiasing Learning and Evaluation
• Estimating individual treatment effect: generalization bounds and algorithms
• Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
• An Optimistic Perspective on Offline Reinforcement Learning
• GenDICE: Generalized Offline Estimation of Stationary Values
• Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model
• Effective Evaluation using Logged Bandit Feedback from Multiple Loggers
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Machine Learning Methods Economists Should Know About
• Measuring Recommender System Effects with Simulated Users
• Counterfactual Estimation and Optimization of Click Metrics for Search Engines
• Doubly robust off-policy evaluation with shrinkage
• Improving offline evaluation of contextual bandit algorithms via bootstrapping techniques
• Counterfactual Reasoning and Learning Systems
• On component interactions in two-stage recommender systems
• Top-K Off-Policy Correction for a REINFORCE Recommender System
• Off-Policy Evaluation of Ranking Policies under Diverse User Behavior
• contextual: Evaluating Contextual Multi-Armed Bandit Problems in R
• Imitation-Regularized Offline Learning
• Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
• CAB: Continuous Adaptive Blending Estimator for Policy Evaluation and Learning
• Evaluation of Explore-Exploit Policies in Multi-result Ranking Systems
• Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• On the Optimality of Batch Policy Optimization Algorithms
• Optimal Off-Policy Evaluation from Multiple Logging Policies
• Cost-Effective Incentive Allocation via Structured Counterfactual Inference
• Towards Content Provider Aware Recommender Systems: A Simulation Study on the Interplay between User and Provider Utilities
• Counterfactual Learning To Rank for Utility-Maximizing Query Autocompletion
• Bayesian Counterfactual Risk Minimization
• Off-policy Learning for Multiple Loggers
• Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Support
• Robust Generalization despite Distribution Shift via Minimum Discriminating Information
• Offline Contextual Bandits for Wireless Network Optimization
• Offline Contextual Bandits with Overparameterized Models
• A Map of Bandits for E-commerce
• Pessimistic Off-Policy Optimization for Learning to Rank