Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks

Abstract

Physics-Informed Neural Networks (PINNs) have proven effective in solving partial differential equations (PDEs), especially when some data are available by seamlessly blending data and physics. However, extending PINNs to high-dimensional and even high-order PDEs encounters significant challenges due to the computational cost associated with automatic differentiation in the residual loss. Herein, we address the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing Hutchinson Trace Estimation (HTE). Starting with the second-order high-dimensional PDEs ubiquitous in scientific computing, HTE transforms the calculation of the entire Hessian matrix into a Hessian vector product (HVP). This approach alleviates the computational bottleneck via Taylor-mode automatic differentiation and significantly reduces memory consumption from the Hessian matrix to HVP. We further showcase HTE's convergence to the original PINN loss and its unbiased behavior under specific conditions. Comparisons with Stochastic Dimension Gradient Descent (SDGD) highlight the distinct advantages of HTE, particularly in scenarios with significant variance among dimensions. We further extend HTE to higher-order and higher-dimensional PDEs, specifically addressing the biharmonic equation. By employing tensor-vector products (TVP), HTE efficiently computes the colossal tensor associated with the fourth-order high-dimensional biharmonic equation, saving memory and enabling rapid computation. The effectiveness of HTE is illustrated through experimental setups, demonstrating comparable convergence rates with SDGD under memory and speed constraints. Additionally, HTE proves valuable in accelerating the Gradient-Enhanced PINN (gPINN) version as well as the Biharmonic equation. Overall, HTE opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs.

arXiv commentPublished in Computer Methods in Applied Mechanics and Engineering

References in corpus (24)

• Solving high-dimensional partial differential equations using deep learning
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• Weak Adversarial Networks for High-dimensional Partial Differential Equations
• Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Generalization in Deep Learning
• Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology
• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• Deep splitting method for parabolic PDEs
• Meta-learning PINN loss functions
• Improved bounds on sample size for implicit matrix trace estimators
• Multilevel Picard iterations for solving smooth semilinear parabolic heat equations
• Solving high-dimensional optimal stopping problems using deep learning
• Overcoming the curse of dimensionality in the numerical approximation of Allen-Cahn partial differential equations via truncated full-history recursive multilevel Picard approximations
• Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations
• Numerical simulations for full history recursive multilevel Picard approximations for systems of high-dimensional partial differential equations
• Overcoming the curse of dimensionality in the numerical approximation of high-dimensional semilinear elliptic partial differential equations
• Is $L^2$ Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?
• How Does Information Bottleneck Help Deep Learning?
• Separable PINN: Mitigating the Curse of Dimensionality in Physics-Informed Neural Networks
• Bias-Variance Trade-off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEs

Cited by in corpus (5)

• Tackling the Curse of Dimensionality with Physics-Informed Neural Networks
• Tackling the Curse of Dimensionality in Fractional and Tempered Fractional PDEs with Physics-Informed Neural Networks
• Separable Physics-informed Neural Networks for Solving the BGK Model of the Boltzmann Equation
• Spectral Informed Neural Network: An Efficient and Low-Memory PINN
• Physics-informed neural networks for solving functional renormalization group on a lattice