Understanding and mitigating gradient pathologies in physics-informed neural networks

Abstract

The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We will also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50-100x across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs}.

28 Pages, 18 figures

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• DGM: A deep learning algorithm for solving partial differential equations
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Understanding Deep Convolutional Networks
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks
• Variational Physics-Informed Neural Networks For Solving Partial Differential Equations
• Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems
• The Mechanics of n-Player Differentiable Games
• Neural Proximal Gradient Descent for Compressive Imaging
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Covariant Compositional Networks For Learning Graphs
• Hidden Fluid Mechanics: A Navier-Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data
• Learning Parameters and Constitutive Relationships with Physics Informed Deep Neural Networks
• Competitive Gradient Descent

Cited by in corpus (67)

• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition
• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain
• A Physics-Informed Machine Learning Approach for Solving Heat Transfer Equation in Advanced Manufacturing and Engineering Applications
• Physics-informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions
• CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• Physics-Informed Neural Network for Modelling the Thermochemical Curing Process of Composite-Tool Systems During Manufacture
• Stiff-PINN: Physics-Informed Neural Network for Stiff Chemical Kinetics
• Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems
• Physics informed neural networks for continuum micromechanics
• PINNeik: Eikonal solution using physics-informed neural networks
• Autonomous Discovery of Unknown Reaction Pathways from Data by Chemical Reaction Neural Network
• Multi-Objective Loss Balancing for Physics-Informed Deep Learning
• Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training
• Physics-Informed Neural Network Method for Forward and Backward Advection-Dispersion Equations
• Physics-informed neural networks for the shallow-water equations on the sphere
• Meta-learning PINN loss functions
• Hybrid FEM-NN models: Combining artificial neural networks with the finite element method
• Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
• When and why PINNs fail to train: A neural tangent kernel perspective
• Deep learning of free boundary and Stefan problems
• Characterizing possible failure modes in physics-informed neural networks
• Physics-informed radial basis network (PIRBN): A local approximating neural network for solving nonlinear PDEs
• A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems
• Explore missing flow dynamics by physics-informed deep learning: the parameterised governing systems
• Learning Unknown Physics of non-Newtonian Fluids
• Physics-informed neural networks (PINNs) for fluid mechanics: A review
• Evolutional Deep Neural Network
• Physics-informed data based neural networks for two-dimensional turbulence
• PI-VAE: Physics-Informed Variational Auto-Encoder for stochastic differential equations
• Choose a Transformer: Fourier or Galerkin
• Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domains
• Multi-Output Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Uncertainties
• PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics
• Error estimates of residual minimization using neural networks for linear PDEs
• ModalPINN: an extension of Physics-Informed Neural Networks with enforced truncated Fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors
• Physics informed deep learning for computational elastodynamics without labeled data
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Physics-aware deep neural networks for surrogate modeling of turbulent natural convection
• Accelerating Simulation of Stiff Nonlinear Systems using Continuous-Time Echo State Networks
• Deep learning for solution and inversion of structural mechanics and vibrations
• Learning Free-Surface Flow with Physics-Informed Neural Networks
• Adapting Physics-Informed Neural Networks to Improve ODE Optimization in Mosquito Population Dynamics
• Hard Encoding of Physics for Learning Spatiotemporal Dynamics
• TensorDiffEq: Scalable Multi-GPU Forward and Inverse Solvers for Physics Informed Neural Networks
• Inverse design of ultracompact multi-focal optical devices by diffractive neural networks
• AdjointNet: Constraining machine learning models with physics-based codes
• NeuralPDE: Automating Physics-Informed Neural Networks (PINNs) with Error Approximations
• Structure-preserving Sparse Identification of Nonlinear Dynamics for Data-driven Modeling
• Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs
• Partition of unity networks: deep hp-approximation
• A deep neural network approach on solving the linear transport model under diffusive scaling
• Modeling Power Systems Dynamics with Symbolic Physics-Informed Neural Networks
• On the Correspondence between Gaussian Processes and Geometric Harmonics
• A Physics-Data-Driven Bayesian Method for Heat Conduction Problems
• Bayesian Hidden Physics Models: Uncertainty Quantification for Discovery of Nonlinear Partial Differential Operators from Data
• Reconstructing a dynamical system and forecasting time series by self-consistent deep learning
• Physics-enhanced Neural Networks in the Small Data Regime
• Transient Stability Analysis with Physics-Informed Neural Networks
• Optimal Transport Based Refinement of Physics-Informed Neural Networks
• Machine Learning for Vortex Induced Vibration in Turbulent Flow
• Deep learning in physics: a study of dielectric quasi-cubic particles in a uniform electric field
• Flow field tomography with uncertainty quantification using a Bayesian physics-informed neural network
• Polynomial-Spline Neural Networks with Exact Integrals