Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding

Abstract

When observed decisions depend only on observed features, off-policy policy evaluation (OPE) methods for sequential decision making problems can estimate the performance of evaluation policies before deploying them. This assumption is frequently violated due to unobserved confounders, unrecorded variables that impact both the decisions and their outcomes. We assess robustness of OPE methods under unobserved confounding by developing worst-case bounds on the performance of an evaluation policy. When unobserved confounders can affect every decision in an episode, we demonstrate that even small amounts of per-decision confounding can heavily bias OPE methods. Fortunately, in a number of important settings found in healthcare, policy-making, operations, and technology, unobserved confounders may primarily affect only one of the many decisions made. Under this less pessimistic model of one-decision confounding, we propose an efficient loss-minimization-based procedure for computing worst-case bounds, and prove its statistical consistency. On two simulated healthcare examples---management of sepsis patients and developmental interventions for autistic children---where this is a reasonable model of confounding, we demonstrate that our method invalidates non-robust results and provides meaningful certificates of robustness, allowing reliable selection of policies even under unobserved confounding.

References in corpus (12)

• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
• Making Contextual Decisions with Low Technical Debt
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models
• Off-Policy Policy Gradient with State Distribution Correction
• Consistent On-Line Off-Policy Evaluation
• Algorithmic Decision Making in the Presence of Unmeasured Confounding
• Combining Parametric and Nonparametric Models for Off-Policy Evaluation
• Learning When-to-Treat Policies

Cited by in corpus (8)

• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• Causal Markov Decision Processes: Learning Good Interventions Efficiently
• Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
• Universal Off-Policy Evaluation
• Towards Robust Off-Policy Evaluation via Human Inputs
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling