Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
arXiv:2102.09907
Abstract
In offline reinforcement learning (RL) an optimal policy is learned solely from a priori collected observational data. However, in observational data, actions are often confounded by unobserved variables. Instrumental variables (IVs), in the context of RL, are the variables whose influence on the state variables is all mediated by the action. When a valid instrument is present, we can recover the confounded transition dynamics through observational data. We study a confounded Markov decision process where the transition dynamics admit an additive nonlinear functional form. Using IVs, we derive a conditional moment restriction through which we can identify transition dynamics based on observational data. We propose a provably efficient IV-aided Value Iteration (IVVI) algorithm based on a primal-dual reformulation of the conditional moment restriction. To our knowledge, this is the first provably efficient algorithm for instrument-aided offline RL.
under review
References in corpus (18)
- SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
- DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
- Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach
- SGD and Hogwild! Convergence Without the Bounded Gradients Assumption
- Deep Generalized Method of Moments for Instrumental Variable Analysis
- Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models
- A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units
- Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
- Deconfounding Reinforcement Learning in Observational Settings
- Confounding-Robust Policy Improvement
- Active Learning for Nonlinear System Identification with Guarantees
- Information Theoretic Regret Bounds for Online Nonlinear Control
- Minimax Estimation of Conditional Moment Models
- Estimating Optimal Treatment Rules with an Instrumental Variable: A Partial Identification Learning Approach
- Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
- Adversarial Generalized Method of Moments
- Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
- Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders