Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks

Abstract

We propose a novel numerical method for high dimensional Hamilton--Jacobi--Bellman (HJB) type elliptic partial differential equations (PDEs). The HJB PDEs, reformulated as optimal control problems, are tackled by the actor-critic framework inspired by reinforcement learning, based on neural network parametrization of the value and control functions. Within the actor-critic framework, we employ a policy gradient approach to improve the control, while for the value function, we derive a variance reduced least-squares temporal difference method using stochastic calculus. To numerically discretize the stochastic control problem, we employ an adaptive step size scheme to improve the accuracy near the domain boundary. Numerical examples up to $20$ spatial dimensions including the linear quadratic regulators, the stochastic Van der Pol oscillators, the diffusive Eikonal equations, and fully nonlinear elliptic PDEs derived from a regulator problem are presented to validate the effectiveness of our proposed method.

23 pages, 4 figures. Add a fully nonlinear example

References in corpus (6)

• Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning
• Deep Learning Approximation for Stochastic Control Problems
• An overview on deep learning-based approximation methods for partial differential equations
• Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approach
• A Derivative-Free Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks
• Convergence of Deep Fictitious Play for Stochastic Differential Games

Cited by in corpus (5)

• Reinforcement learning for traversing chemical structure space: Optimizing transition states and minimum energy paths of molecules
• Solving non-linear Kolmogorov equations in large dimensions by using deep learning: a numerical comparison of discretization schemes
• Improving control based importance sampling strategies for metastable diffusions via adapted metadynamics
• Lax-Oleinik-type formulas and efficient algorithms for certain high-dimensional optimal control problems
• Hopf-type representation formulas and efficient algorithms for certain high-dimensional optimal control problems