Optimal Baseline Corrections for Off-Policy Contextual Bandits

Abstract

The off-policy learning paradigm allows for recommender systems and general ranking applications to be framed as decision-making problems, where we aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. With unbiasedness comes potentially high variance, and prevalent methods exist to reduce estimation variance. These methods typically make use of control variates, either additive (i.e., baseline corrections or doubly robust methods) or multiplicative (i.e., self-normalisation). Our work unifies these approaches by proposing a single framework built on their equivalence in learning scenarios. The foundation of our framework is the derivation of an equivalent baseline correction for all of the existing control variates. Consequently, our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it. This optimal estimator brings significantly improved performance in both evaluation and learning, and minimizes data requirements. Empirical observations corroborate our theoretical findings.

References in corpus (5)

• Doubly Robust Policy Evaluation and Optimization
• Carousel Personalization in Music Streaming Apps with Contextual Bandits
• Evaluating the Robustness of Off-Policy Evaluation
• BLOB : A Probabilistic Model for Recommendation that Combines Organic and Bandit Signals
• Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimization

Cited by in corpus (4)

• Exploitation Over Exploration: Unmasking the Bias in Linear Bandit Recommender Offline Evaluation
• Meta Off-Policy Estimation
• Towards Two-Stage Counterfactual Learning to Rank
• Exposure-Based Reinforcement Learning to Rank