Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks

Abstract

We propose a novel numerical method for high dimensional Hamilton--Jacobi--Bellman (HJB) type elliptic partial differential equations (PDEs). The HJB PDEs, reformulated as optimal control problems, are tackled by the actor-critic framework inspired by reinforcement learning, based on neural network parametrization of the value and control functions. Within the actor-critic framework, we employ a policy gradient approach to improve the control, while for the value function, we derive a variance reduced least-squares temporal difference method using stochastic calculus. To numerically discretize the stochastic control problem, we employ an adaptive step size scheme to improve the accuracy near the domain boundary. Numerical examples up to $20$ spatial dimensions including the linear quadratic regulators, the stochastic Van der Pol oscillators, the diffusive Eikonal equations, and fully nonlinear elliptic PDEs derived from a regulator problem are presented to validate the effectiveness of our proposed method.

arXiv comment23 pages, 4 figures. Add a fully nonlinear example

References in corpus (16)

• Playing Atari with Deep Reinforcement Learning
• Solving high-dimensional partial differential equations using deep learning
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Sample Efficient Actor-Critic with Experience Replay
• Algorithms for Solving High Dimensional PDEs: From Nonlinear Monte Carlo to Machine Learning
• Convergence of the Deep BSDE Method for Coupled FBSDEs
• Deep Learning Approximation for Stochastic Control Problems
• Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equations
• Polynomial approximation of high-dimensional Hamilton-Jacobi-Bellman equations and applications to feedback control of semilinear parabolic PDEs
• An overview on deep learning-based approximation methods for partial differential equations
• Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approach
• A Derivative-Free Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks
• Off-Policy Actor-Critic
• Learning Deep Stochastic Optimal Control Policies using Forward-Backward SDEs
• Deep Fictitious Play for Finding Markovian Nash Equilibrium in Multi-Agent Games
• Convergence of Deep Fictitious Play for Stochastic Differential Games

Cited by in corpus (5)

• Reinforcement learning for traversing chemical structure space: Optimizing transition states and minimum energy paths of molecules
• Solving non-linear Kolmogorov equations in large dimensions by using deep learning: a numerical comparison of discretization schemes
• Improving control based importance sampling strategies for metastable diffusions via adapted metadynamics
• Lax-Oleinik-type formulas and efficient algorithms for certain high-dimensional optimal control problems
• Hopf-type representation formulas and efficient algorithms for certain high-dimensional optimal control problems