When and why PINNs fail to train: A neural tangent kernel perspective

Abstract

Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at \url{https://github.com/PredictiveIntelligenceLab/PINNsNTK}.

29 pages, 8 figures

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• DeepXDE: A deep learning library for solving differential equations
• DGM: A deep learning algorithm for solving partial differential equations
• Solving high-dimensional partial differential equations using deep learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• Deep Neural Networks as Gaussian Processes
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• On the Spectral Bias of Neural Networks
• Deep learning of free boundary and Stefan problems
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Learning Parameters and Constitutive Relationships with Physics Informed Deep Neural Networks
• Physics-informed deep generative models
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• Physics-Constrained Bayesian Neural Network for Fluid Flow Reconstruction with Sparse and Noisy Data

Cited by in corpus (38)

• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Uncovering near-wall blood flow from sparse data with physics-informed neural networks
• Polyconvex anisotropic hyperelasticity with neural networks
• PINNeik: Eikonal solution using physics-informed neural networks
• Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training
• Physics-Informed Neural Operator for Learning Partial Differential Equations
• Physics-informed neural networks for the shallow-water equations on the sphere
• Hybrid FEM-NN models: Combining artificial neural networks with the finite element method
• Characterizing possible failure modes in physics-informed neural networks
• Physics-informed neural networks (PINNs) for fluid mechanics: A review
• Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domains
• A Physics-Informed Neural Network approach for compartmental epidemiological models
• On quadrature rules for solving Partial Differential Equations using Neural Networks
• Error estimates of residual minimization using neural networks for linear PDEs
• ModalPINN: an extension of Physics-Informed Neural Networks with enforced truncated Fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors
• Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Solving and Learning Nonlinear PDEs with Gaussian Processes
• Structure Probing Neural Network Deflation
• Adapting Physics-Informed Neural Networks to Improve ODE Optimization in Mosquito Population Dynamics
• Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networks
• TensorDiffEq: Scalable Multi-GPU Forward and Inverse Solvers for Physics Informed Neural Networks
• A Priori Analysis of Stable Neural Network Solutions to Numerical PDEs
• Reproducing Activation Function for Deep Learning
• Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs
• Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks
• Fast PDE-constrained optimization via self-supervised operator learning
• A deep neural network approach on solving the linear transport model under diffusive scaling
• Sinc Kolmogorov-Arnold network and its application for solving PDEs with singularities
• Analysis of Deep Ritz Methods for Laplace Equations with Dirichlet Boundary Conditions
• BINet: Learning to Solve Partial Differential Equations with Boundary Integral Networks
• Physics-enhanced Neural Networks in the Small Data Regime
• Optimal Transport Based Refinement of Physics-Informed Neural Networks
• Reconstructing a dynamical system and forecasting time series by self-consistent deep learning
• Deep learning in physics: a study of dielectric quasi-cubic particles in a uniform electric field
• Flow field tomography with uncertainty quantification using a Bayesian physics-informed neural network
• Investigating a New Approach to Quasinormal Modes: Physics-Informed Neural Networks