Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models

Abstract

We introduce an off-policy evaluation procedure for highlighting episodes where applying a reinforcement learned (RL) policy is likely to have produced a substantially different outcome than the observed policy. In particular, we introduce a class of structural causal models (SCMs) for generating counterfactual trajectories in finite partially observable Markov Decision Processes (POMDPs). We see this as a useful procedure for off-policy "debugging" in high-risk settings (e.g., healthcare); by decomposing the expected difference in reward between the RL and observed policy into specific episodes, we can identify episodes where the counterfactual difference in reward is most dramatic. This in turn can be used to facilitate review of specific episodes by domain experts. We demonstrate the utility of this procedure with a synthetic environment of sepsis management.

To appear in ICML 2019

Cited by in corpus (31)

• Deep Structural Causal Models for Tractable Counterfactual Inference
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
• POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning
• Neuropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluation
• Off-Policy Evaluation and Learning for External Validity under a Covariate Shift
• Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions
• MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming
• Causal Modeling for Fairness in Dynamical Systems
• Bootstrapping Fitted Q-Evaluation for Off-Policy Inference
• Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models
• Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• To Measure or Not: A Cost-Sensitive, Selective Measuring Environment for Agricultural Management Decisions with Reinforcement Learning
• Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation
• Counterfactual Explanations in Sequential Decision Making Under Uncertainty
• Triply Robust Off-Policy Evaluation
• A Practical Guide of Off-Policy Evaluation for Bandit Problems
• A Spectral Approach to Off-Policy Evaluation for POMDPs
• Challenges for Reinforcement Learning in Healthcare
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
• Reinforcement Learning with Efficient Active Feature Acquisition
• Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studies
• Off-Policy Evaluation of Bandit Algorithm from Dependent Samples under Batch Update Policy
• Counterfactual Off-Policy Training for Neural Response Generation
• Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation
• Learning Generalized Gumbel-max Causal Mechanisms
• Learning Under Adversarial and Interventional Shifts
• The Adaptive Doubly Robust Estimator for Policy Evaluation in Adaptive Experiments and a Paradox Concerning Logging Policy
• Fast Generating A Large Number of Gumbel-Max Variables