Operator Splitting, Policy Iteration, and Machine Learning for Stochastic Optimal Control

Abstract

We propose a splitting approach to solve the second-order Hamilton--Jacobi equation, reducing it to a heat step and a purely first-order step. The latter is implemented using a gradient value policy iteration algorithm, enabling efficient characteristic-based machine learning methods. We establish convergence rates for the splitting method. In particular, with $h$ the splitting step, the $L^{\infty }$ error is bounded between $\mathcal{O}(h)$ and $\mathcal{O}(h^{1/5})$ for Lipschitz data, improving to $\mathcal{O}(h^{1/3})$ for semiconcave data. In the periodic setting, we also obtain an $L^1$ error of order $\mathcal{O}(h^{1/2})$. For the first-order step, we provide a weighted $L^2$ error analysis that shows exponential convergence. Each iteration solves linear characteristic equations and learns the value function by minimizing a weighted value gradient loss. The approach yields stable and accurate numerical results.

37 pages, with improved results for Theorem 1.1