Off-policy Learning for Multiple Loggers
arXiv:1907.09652
Abstract
It is well known that the historical logs are used for evaluating and learning policies in interactive systems, e.g. recommendation, search, and online advertising. Since direct online policy learning usually harms user experiences, it is more crucial to apply off-policy learning in real-world applications instead. Though there have been some existing works, most are focusing on learning with one single historical policy. However, in practice, usually a number of parallel experiments, e.g. multiple AB tests, are performed simultaneously. To make full use of such historical data, learning policies from multiple loggers becomes necessary. Motivated by this, in this paper, we investigate off-policy learning when the training data coming from multiple historical policies. Specifically, policies, e.g. neural networks, can be learned directly from multi-logger data, with counterfactual estimators. In order to understand the generalization ability of such estimator better, we conduct generalization error analysis for the empirical risk minimization problem. We then introduce the generalization error bound as the new risk function, which can be reduced to a constrained optimization problem. Finally, we give the corresponding learning algorithm for the new constrained problem, where we can appeal to the minimax problems to control the constraints. Extensive experiments on benchmark datasets demonstrate that the proposed methods achieve better performances than the state-of-the-arts.
References in corpus (14)
- Scikit-learn: Machine Learning in Python
- Categorical Reparameterization with Gumbel-Softmax
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
- Deep Reinforcement Learning for Page-wise Recommendations
- Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
- Doubly Robust Policy Evaluation and Learning
- Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
- Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
- Learning from Logged Implicit Exploration Data
- Deep Reinforcement Learning for List-wise Recommendations
- Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems
- Counterfactual Estimation and Optimization of Click Metrics for Search Engines