Asymptotic Theory for IV-Based Reinforcement Learning with Potential Endogeneity
arXiv:2103.04021
Abstract
In the standard data analysis framework, data is collected (once and for all), and then data analysis is carried out. However, with the advancement of digital technology, decision-makers constantly analyze past data and generate new data through their decisions. We model this as a Markov decision process and show that the dynamic interaction between data generation and data analysis leads to a new type of bias -- reinforcement bias -- that exacerbates the endogeneity problem in standard data analysis. We propose a class of instrument variable (IV)-based reinforcement learning (RL) algorithms to correct for the bias and establish their theoretical properties by incorporating them into a stochastic approximation (SA) framework. Our analysis accommodates iterate-dependent Markovian structures and, therefore, can be used to study RL algorithms with policy improvement. We also provide formulas for inference on optimal policies of the IV-RL algorithms. These formulas highlight how intertemporal dependency of the Markovian environment affects the inference.
References in corpus (8)
- An Introduction to Deep Reinforcement Learning
- Convergence rate of linear two-time-scale stochastic approximation
- Finite-Sample Analysis for SARSA with Linear Function Approximation
- Convergence rate and averaging of nonlinear two-time-scale stochastic approximation algorithms
- Instrumental Variables: An Econometrician's Perspective
- Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning
- Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noise
- Finite-Time Performance Bounds and Adaptive Learning Rate Selection for Two Time-Scale Reinforcement Learning