On Instrumental Variable Regression for Deep Offline Policy Evaluation

Abstract

We show that the popular reinforcement learning (RL) strategy of estimating the state-action value (Q-function) by minimizing the mean squared Bellman error leads to a regression problem with confounding, the inputs and output noise being correlated. Hence, direct minimization of the Bellman error can result in significantly biased Q-function estimates. We explain why fixing the target Q-network in Deep Q-Networks and Fitted Q Evaluation provides a way of overcoming this confounding, thus shedding new light on this popular but not well understood trick in the deep RL literature. An alternative approach to address confounding is to leverage techniques developed in the causality literature, notably instrumental variables (IV). We bring together here the literature on IV and RL by investigating whether IV approaches can lead to improved Q-function estimates. This paper analyzes and compares a wide range of recent IV methods in the context of offline policy evaluation (OPE), where the goal is to estimate the value of a policy using logged data only. By applying different IV techniques to OPE, we are not only able to recover previously proposed OPE methods such as model-based techniques but also to obtain competitive new techniques. We find empirically that state-of-the-art OPE methods are closely matched in performance by some IV methods such as AGMM, which were not developed for OPE. We open-source all our code and datasets at https://github.com/liyuan9988/IVOPEwithACME.

arXiv commentAccepted by Journal of Machine Learning Research in 11/2022

References in corpus (21)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Distributed Distributional Deterministic Policy Gradients
• Doubly Robust Policy Evaluation and Learning
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Optimal Sup-norm Rates and Uniform Inference on Nonlinear Functionals of Nonparametric IV Regression
• Acme: A Research Framework for Distributed Reinforcement Learning
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Deep Generalized Method of Moments for Instrumental Variable Analysis
• Benchmarks for Deep Off-Policy Evaluation
• Off-Policy Evaluation via the Regularized Lagrangian
• Minimax Estimation of Conditional Moment Models
• Adversarial Generalized Method of Moments
• Dual Instrumental Variable Regression
• Learning Deep Features in Instrumental Variable Regression
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach
• Asymptotic Theory for IV-Based Reinforcement Learning with Potential Endogeneity