Efficient training of physics-informed neural networks via importance sampling

Abstract

Physics-Informed Neural Networks (PINNs) are a class of deep neural networks that are trained, using automatic differentiation, to compute the response of systems governed by partial differential equations (PDEs). The training of PINNs is simulation-free, and does not require any training dataset to be obtained from numerical PDE solvers. Instead, it only requires the physical problem description, including the governing laws of physics, domain geometry, initial/boundary conditions, and the material properties. This training usually involves solving a non-convex optimization problem using variants of the stochastic gradient descent method, with the gradient of the loss function approximated on a batch of collocation points, selected randomly in each iteration according to a uniform distribution. Despite the success of PINNs in accurately solving a wide variety of PDEs, the method still requires improvements in terms of computational efficiency. To this end, in this paper, we study the performance of an importance sampling approach for efficient training of PINNs. Using numerical examples together with theoretical evidences, we show that in each training iteration, sampling the collocation points according to a distribution proportional to the loss function will improve the convergence behavior of the PINNs training. Additionally, we show that providing a piecewise constant approximation to the loss function for faster importance sampling can further improve the training efficiency. This importance sampling approach is straightforward and easy to implement in the existing PINN codes, and also does not introduce any new hyperparameter to calibrate. The numerical examples include elasticity, diffusion and plane stress problems, through which we numerically verify the accuracy and efficiency of the importance sampling approach compared to the predominant uniform sampling approach.

arXiv commentComputer-Aided Civil and Infrastructure Engineering (2021)

References in corpus (27)

• TensorFlow: A system for large-scale machine learning
• ADADELTA: An Adaptive Learning Rate Method
• An overview of gradient descent optimization algorithms
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• DGM: A deep learning algorithm for solving partial differential equations
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems
• A unified deep artificial neural network approach to partial differential equations in complex geometries
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• A Deep Collocation Method for the Bending Analysis of Kirchhoff Plate
• Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations
• Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• Deep Learning of Vortex Induced Vibrations
• Data Driven Governing Equations Approximation Using Deep Neural Networks
• PDE-Net: Learning PDEs from Data
• Deep Learning for Accelerated Reliability Analysis of Infrastructure Networks
• A Deep Neural Network Surrogate for High-Dimensional Random Partial Differential Equations
• Physics-Driven Regularization of Deep Neural Networks for Enhanced Engineering Design and Analysis
• Variance Reduction in SGD by Distributed Importance Sampling
• Hidden Fluid Mechanics: A Navier-Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data
• Biased Importance Sampling for Deep Neural Network Training
• The Neural Network Approach to Inverse Problems in Differential Equations
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Deep convolutional neural networks for uncertainty propagation in random fields

Cited by in corpus (31)

• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
• CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method
• Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarios
• Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training
• Deep Learning in Deterministic Computational Mechanics
• Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goal
• Physics-informed neural network solution of thermo-hydro-mechanical (THM) processes in porous media
• $\Delta$-PINNs: physics-informed neural networks on complex geometries
• A solver for subsonic flow around airfoils based on physics-informed neural networks and mesh transformation
• Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks
• Physics-Informed Neural Networks for Parametric Compressible Euler Equations
• GAS: A Gaussian Mixture Distribution-Based Adaptive Sampling Method for PINNs
• Neural Eikonal Solver: improving accuracy of physics-informed neural networks for solving eikonal equation in case of caustics
• Adapting Physics-Informed Neural Networks to Improve ODE Optimization in Mosquito Population Dynamics
• A multifidelity approach to continual learning for physical systems
• Machine Learning in Acoustics: A Review and Open-Source Repository
• LSA-PINN: Linear Boundary Connectivity Loss for Solving PDEs on Complex Geometry
• Scientific machine learning in Hydrology: a unified perspective
• Discovering Physics-Informed Neural Networks Model for Solving Partial Differential Equations through Evolutionary Computation
• Training Deep Physics-Informed Kolmogorov-Arnold Networks
• Investigating Guiding Information for Adaptive Collocation Point Sampling in PINNs
• Learning Feynman integrals from differential equations with neural networks
• Learning quantities of interest from parametric PDEs: An efficient neural-weighted Minimal Residual approach
• A composable autoencoder-based iterative algorithm for accelerating numerical simulations
• Constrained or Unconstrained? Neural-Network-Based Equation Discovery from Data
• Fixed-budget online adaptive learning for physics-informed neural networks. Towards parameterized problem inference
• Active learning with physics-informed neural networks for optimal sensor placement in deep tunneling through transversely isotropic elastic rocks
• Leveraging Influence Functions for Resampling Data in Physics-Informed Neural Networks
• Quasi-Random Physics-informed Neural Networks
• NuGraph2 with Context-Aware Inputs: Physics-Inspired Improvements in Semantic Segmentation