Doubly Robust Policy Evaluation and Learning

Abstract

We study decision making in environments where the reward is only partially observed, but can be modeled as a function of an action and an observed context. This setting, known as contextual bandits, encompasses a wide variety of applications including health-care policy and Internet advertising. A central task is evaluation of a new policy given historic data consisting of contexts, actions and received rewards. The key challenge is that the past data typically does not faithfully represent proportions of actions taken by a new policy. Previous approaches rely either on models of rewards or models of the past policy. The former are plagued by a large bias whereas the latter have a large variance. In this work, we leverage the strength and overcome the weaknesses of the two approaches by applying the doubly robust technique to the problems of policy evaluation and optimization. We prove that this approach yields accurate value estimates when we have either a good (but not necessarily consistent) model of rewards or a good (but not necessarily consistent) model of past policy. Extensive empirical comparison demonstrates that the doubly robust approach uniformly improves over existing techniques, achieving both lower variance in value estimation and better policies. As such, we expect the doubly robust approach to become common practice.

Published at ICML 2011, 8 pages, 6 figures

Cited by in corpus (137)

• Recursive Partitioning for Heterogeneous Causal Effects
• Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
• Challenges of Real-World Reinforcement Learning
• Learning Representations for Counterfactual Inference
• Recommendations as Treatments: Debiasing Learning and Evaluation
• Doubly Robust Policy Evaluation and Optimization
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Double Machine Learning based Program Evaluation under Unconfoundedness
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Quasi-Oracle Estimation of Heterogeneous Treatment Effects
• Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• A Survey on Contextual Multi-armed Bandits
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• Making Contextual Decisions with Low Technical Debt
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Causal Decision Trees
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Enhanced Doubly Robust Learning for Debiasing Post-click Conversion Rate Estimation
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Intervention Harvesting for Context-Dependent Examination-Bias Estimation
• A Contextual Bandit Bake-off
• GenDICE: Generalized Offline Estimation of Stationary Values
• Practical Contextual Bandits with Regression Oracles
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• Adaptive Metamorphic Testing with Contextual Bandits
• Off-policy evaluation for slate recommendation
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Sequential Batch Learning in Finite-Action Linear Contextual Bandits
• Information Theoretic Counterfactual Learning from Missing-Not-At-Random Feedback
• Large-scale Validation of Counterfactual Learning Methods: A Test-Bed
• Estimation Considerations in Contextual Bandits
• Machine Learning Methods Economists Should Know About
• A Survey on Causal Inference
• Off-Policy Evaluation via Off-Policy Classification
• Bridging the gap: Towards an Expanded Toolkit for AI-driven Decision-Making in the Public Sector
• Multi-step Off-policy Learning Without Importance Sampling Ratios
• Causal Etiology of the Research of James M. Robins
• Optimal and Adaptive Off-policy Evaluation in Contextual Bandits
• Counterfactual Estimation and Optimization of Click Metrics for Search Engines
• Improving offline evaluation of contextual bandit algorithms via bootstrapping techniques
• Doubly robust off-policy evaluation with shrinkage
• Algorithmic Decision Making in the Presence of Unmeasured Confounding
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
• Counterfactual Reasoning and Learning Systems
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
• Off-Policy Evaluation and Learning for External Validity under a Covariate Shift
• Causal Modeling for Fairness in Dynamical Systems
• Off-Policy Risk Assessment in Contextual Bandits
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning
• On component interactions in two-stage recommender systems
• Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting
• Conservative Policy Construction Using Variational Autoencoders for Logged Data with Missing Values
• Fair Decisions Despite Imperfect Predictions
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• contextual: Evaluating Contextual Multi-Armed Bandit Problems in R
• Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning
• Learning When-to-Treat Policies
• Offline Policy Selection under Uncertainty
• Representation Balancing MDPs for Off-Policy Policy Evaluation
• Sequential Evaluation and Generation Framework for Combinatorial Recommender System
• Task Selection Policies for Multitask Learning
• Importance Sampling Policy Evaluation with an Estimated Behavior Policy
• A nonparametric Bayesian analysis of heterogeneous treatment effects in digital experimentation
• Efficient Policy Learning from Surrogate-Loss Classification Reductions
• Policy Evaluation with Latent Confounders via Optimal Balance
• Reinforcement Learning for Uplift Modeling
• Efficient Contextual Bandits with Continuous Actions
• Counterfactual Learning for Machine Translation: Degeneracies and Solutions
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Learning Optimal Policies from Observational Data
• Bandit Algorithms for Precision Medicine
• Accurate Inference for Adaptive Linear Models
• Imitation-Regularized Offline Learning
• Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation
• CAB: Continuous Adaptive Blending Estimator for Policy Evaluation and Learning
• Empirical Likelihood for Contextual Bandits
• P3O: Policy-on Policy-off Policy Optimization
• Asymmetric Tri-training for Debiasing Missing-Not-At-Random Explicit Feedback
• COVID-19: Strategies for Allocation of Test Kits
• Combining Offline Causal Inference and Online Bandit Learning for Data Driven Decision
• Permutation Weighting
• Counterfactual Cross-Validation: Stable Model Selection Procedure for Causal Inference Models
• Gaussian Process Mixtures for Estimating Heterogeneous Treatment Effects
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• Balanced Linear Contextual Bandits
• Off-policy Confidence Sequences
• Computational Causal Inference
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• Loss Functions for Discrete Contextual Pricing with Observational Data
• Double Robust Representation Learning for Counterfactual Prediction
• Learning Individualized Treatment Rules with Estimated Translated Inverse Propensity Score
• Cost-Effective Incentive Allocation via Structured Counterfactual Inference
• Discretizing Logged Interaction Data Biases Learning for Decision-Making
• Offline Evaluation of Ranking Policies with Click Models
• Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth) Interventions
• Personalized Advertisement Recommendation: A Ranking Approach to Address the Ubiquitous Click Sparsity Problem
• Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning
• Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation
• Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networks
• Control Variates for Slate Off-Policy Evaluation
• A framework for massive scale personalized promotion
• Data-Driven Off-Policy Estimator Selection: An Application in User Marketing on An Online Content Delivery Service
• Bayesian Counterfactual Risk Minimization
• Off-Policy Evaluation of Slate Policies under Bayes Risk
• Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probability
• Balancing Method for High Dimensional Causal Inference
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Robust Contextual Bandit via the Capped-${\ell }_2$ norm
• Markov Decision Process modeled with Bandits for Sequential Decision Making in Linear-flow
• Off-policy Learning for Multiple Loggers
• Privacy-Preserving Causal Inference via Inverse Probability Weighting
• Supervised Off-Policy Ranking
• Non-Stationary Off-Policy Optimization
• Doubly Robust Direct Learning for Estimating Conditional Average Treatment Effect
• Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation
• On the Fairness of Randomized Trials for Recommendation with Heterogeneous Demographics and Beyond
• Offline Contextual Bandits with Overparameterized Models
• The Adaptive Doubly Robust Estimator for Policy Evaluation in Adaptive Experiments and a Paradox Concerning Logging Policy
• Balanced Policy Evaluation and Learning for Right Censored Data
• Robust Counterfactual Inferences using Feature Learning and their Applications
• Discovering Context Specific Causal Relationships
• Unbiased Estimation of the Value of an Optimized Policy
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation
• Active Offline Policy Selection
• Stochastic Doubly Robust Gradient
• Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Bandits
• A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning
• Stochastic Intervention for Causal Inference via Reinforcement Learning
• Doubly Robust Off-Policy Learning on Low-Dimensional Manifolds by Deep Neural Networks
• Robust Bandit Learning with Imperfect Context
• GEAR: On Optimal Decision Making with Auxiliary Data
• Learning Robust Decision Policies from Observational Data
• Large-scale Causal Approaches to Debiasing Post-click Conversion Rate Estimation with Multi-task Learning