Average Cost Optimality Inequality for Markov Decision Processes with Borel Spaces and Universally Measurable Policies

Abstract

We consider average-cost Markov decision processes (MDPs) with Borel state and action spaces and universally measurable policies. For the nonnegative cost model and an unbounded cost model with a Lyapunov-type stability character, we introduce a set of new conditions under which we prove the average cost optimality inequality (ACOI) via the vanishing discount factor approach. Unlike most existing results on the ACOI, our result does not require any compactness and continuity conditions on the MDPs. Instead, the main idea is to use the almost-uniform-convergence property of a pointwise convergent sequence of measurable functions as asserted in Egoroff's theorem. Our conditions are formulated in order to exploit this property. Among others, we require that for each state, on selected subsets of actions at that state, the state transition stochastic kernel is majorized by finite measures. We combine this majorization property of the transition kernel with Egoroff's theorem to prove the ACOI.

arXiv comment34 pages; to appear in SIAM Journal on Control and Optimization (this is the accepted version before the galley proof). The contents of this paper consist of (i) the author's results given previously in Section 3 of arXiv:1901.03374v1 and (ii) additional sections for an extended discussion and illustrative examples. arXiv admin note: text overlap with arXiv:1901.03374

References in corpus (4)

• On Convergence of Value Iteration for a Class of Total Cost Markov Decision Processes
• On the Minimum Pair Approach for Average-Cost Markov Decision Processes with Countable Discrete Action Spaces and Strictly Unbounded Costs
• Fatou's Lemma in Its Classic Form and Lebesgue's Convergence Theorems for Varying Measures with Applications to MDPs
• On Markov Decision Processes with Borel Spaces and an Average Cost Criterion

Cited by in corpus (3)

• On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs
• Average-Cost Optimality Results for Borel-Space Markov Decision Processes with Universally Measurable Policies
• On Strategic Measures and Optimality Properties in Discrete-Time Stochastic Control with Universally Measurable Policies