Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making
arXiv:2002.01751
Abstract
The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an important role for identifying the optimal policy in high-order Markov decision processes and partially observable MDPs. We apply our test to both synthetic datasets and a real data example from mobile health studies to illustrate its usefulness.
References in corpus (8)
- Basic Properties of Strong Mixing Conditions. A Survey and Some Open Questions
- Estimating high-dimensional directed acyclic graphs with the PC-algorithm
- Kernel-based Conditional Independence Test and Application in Causal Discovery
- Optimal Uniform Convergence Rates and Asymptotic Normality for Series Estimators Under Weak Dependence and Weak Conditions
- Exponential inequalities for self-normalized martingales with applications
- Finite-Sample Analysis for SARSA with Linear Function Approximation
- Detailed proof of Nazarov's inequality
- A high dimensional Central Limit Theorem for martingales, with applications to context tree models