Policy Iteration for Stationary Discounted Hamilton--Jacobi--Bellman Equations: A Viscosity Approach

Abstract

We study policy iteration (PI) for deterministic infinite-horizon discounted control problems characterized by stationary Hamilton--Jacobi--Bellman equations. For general viscosity solutions, the classical gradient-based policy improvement step need not be defined pointwise. We introduce a semi-discrete formulation with centered difference quotients at scale $h$ and a separate artificial-viscosity term of order $O(h)$. The resulting stencil is monotone, and the positive discount yields a resolvent contraction. Under bounded Lipschitz data and a globally Lipschitz minimizing policy map, we prove monotone and geometric convergence of the value iterates for each fixed $h>0$, together with a local quadratic estimate whose constant is of order $h^{-2}$. Under the additional condition $λ>\Lip_x(f)$, we establish $\Vert V^h-V{\Vert }_{\infty }\le C\sqrt{h}$ and combine the discretization and iteration errors into a quantitative bound. A bounded Lipschitz example shows that the $\sqrt{h}$ exponent is sharp for this scheme. The combined estimate gives a sufficient iteration count of order $h^{-1}\log (1/h)$ to attain an error of order $\sqrt{h}$. In bounded-domain experiments, the smooth one-dimensional benchmark exhibits the predicted discretization plateau, while a nonlinear two-dimensional manufactured benchmark isolates convergence to the discrete solution. Exact policy evaluation gives substantially faster local convergence than the global geometric bound. A neural evaluation diagnostic illustrates the importance of controlling boundary errors as well as interior residuals.