Ergodic Theory for Controlled Markov Chains with Stationary Inputs

Abstract

Consider a stochastic process $\{X(t)\}$ on a finite state space $\mathsf{X}=\{1,\dots ,d\}$. It is conditionally Markov, given a real-valued `input process' $\{\zeta (t)\}$. This is assumed to be small, which is modeled through the scaling, \[ ζ_t = \varepsilon ζ^1_t, \qquad 0\le \varepsilon \le 1\,, \] where $\{{\zeta }^1(t)\}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\{\zeta (t)\}$: (i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\{X^{\bullet }$(t)\}obtained with $\{\zeta (t)\}\equiv 0$. The triple $(\{X(t)\},\{X^{\bullet }(t)\},\{\zeta (t)\})$ is a jointly stationary process satisfying \[ {\sf P}\{X(t) \neq X^\bullet(t)\} = O(\varepsilon) \] Moreover, a second-order Taylor-series approximation is obtained: \[ {\sf P}\{X(t) =i \} ={\sf P}\{X^\bullet(t) =i \} + \varepsilon^2 \varrho(i) + o(\varepsilon^2),\quad 1\le i\le d, \] with an explicit formula for the vector $\varrho \in {\mathbb{R}}^d$. (ii) For any $m\ge 1$ and any function $f:\{1,\dots ,d\}\times \mathbb{R}\to {\mathbb{R}}^m$, the stationary stochastic process $Y(t)=f(X(t),\zeta (t))$ has a power spectral density ${\text{S}}_f$ that admits a second order Taylor series expansion: A function ${\text{S}}_f^{(2)}:[-\pi ,\pi ]\to {\mathbb{C}}^{m\times m}$ is constructed such that \[ \text{S}_f(θ) = \text{S}^\bullet_f(θ) + \varepsilon^2 \text{S}_f^{(2)}(θ) + o(\varepsilon^2),\quad θ\in [-π,π] . \] An explicit formula for the function ${\text{S}}_f^{(2)}$ is obtained, based in part on the bounds in (i). The results are illustrated using a version of the timing channel of Anantharam and Verdu.

References in corpus (3)

• Ancillary Service to the Grid Using Intelligent Deferrable Loads
• State Estimation for the Individual and the Population in Mean Field Control with Application to Demand Dispatch
• Individual risk in mean-field control models for decentralized control, with application to automated demand response