Two Timescale Stochastic Approximation with Controlled Markov noise and Off-policy temporal difference learning

Prasenjit Karmakar, Shalabh Bhatnagar

We present for the first time an asymptotic convergence analysis of two time-scale stochastic approximation driven by `controlled' Markov noise. In particular, both the faster and…