Understanding and mitigating gradient pathologies in physics-informed neural networks

Abstract

The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We will also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50-100x across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs}.

28 Pages, 18 figures

References in corpus (4)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Variational Physics-Informed Neural Networks For Solving Partial Differential Equations
• Covariant Compositional Networks For Learning Graphs

Cited by in corpus (21)

• When and why PINNs fail to train: A neural tangent kernel perspective
• Physics-informed neural networks (PINNs) for fluid mechanics: A review
• Physics-informed data based neural networks for two-dimensional turbulence
• PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics
• Physics informed deep learning for computational elastodynamics without labeled data
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Physics-aware deep neural networks for surrogate modeling of turbulent natural convection
• TensorDiffEq: Scalable Multi-GPU Forward and Inverse Solvers for Physics Informed Neural Networks
• AdjointNet: Constraining machine learning models with physics-based codes
• NeuralPDE: Automating Physics-Informed Neural Networks (PINNs) with Error Approximations
• Structure-preserving Sparse Identification of Nonlinear Dynamics for Data-driven Modeling
• Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs
• Partition of unity networks: deep hp-approximation
• A deep neural network approach on solving the linear transport model under diffusive scaling
• A Physics-Data-Driven Bayesian Method for Heat Conduction Problems
• On the Correspondence between Gaussian Processes and Geometric Harmonics
• Bayesian Hidden Physics Models: Uncertainty Quantification for Discovery of Nonlinear Partial Differential Operators from Data
• Optimal Transport Based Refinement of Physics-Informed Neural Networks
• Reconstructing a dynamical system and forecasting time series by self-consistent deep learning
• Machine Learning for Vortex Induced Vibration in Turbulent Flow
• Polynomial-Spline Neural Networks with Exact Integrals