Likelihood Ratio Gradient Estimation for Steady-State Parameters

Abstract

We consider a discrete-time Markov chain $\mathbf{\Phi }$ on a general state-space $\mathsf{X}$, whose transition probabilities are parameterized by a real-valued vector $\mathbf{\theta }$. Under the assumption that $\mathbf{\Phi }$ is geometrically ergodic with corresponding stationary distribution $\pi (\mathbf{\theta })$, we are interested in estimating the gradient $\nabla \alpha (\mathbf{\theta })$ of the steady-state expectation $$\alpha (\mathbf{\theta })=\pi (\mathbf{\theta })f.$$ To this end, we first give sufficient conditions for the differentiability of $\alpha (\mathbf{\theta })$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.