Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)

Abstract

This paper explores the difficulties in solving partial differential equations (PDEs) using physics-informed neural networks (PINNs). PINNs use physics as a regularization term in the objective function. However, a drawback of this approach is the requirement for manual hyperparameter tuning, making it impractical in the absence of validation data or prior knowledge of the solution. Our investigations of the loss landscapes and backpropagated gradients in the presence of physics reveal that existing methods produce non-convex loss landscapes that are hard to navigate. Our findings demonstrate that high-order PDEs contaminate backpropagated gradients and hinder convergence. To address these challenges, we introduce a novel method that bypasses the calculation of high-order derivative operators and mitigates the contamination of backpropagated gradients. Consequently, we reduce the dimension of the search space and make learning PDEs with non-smooth solutions feasible. Our method also provides a mechanism to focus on complex regions of the domain. Besides, we present a dual unconstrained formulation based on Lagrange multiplier method to enforce equality constraints on the model's prediction, with adaptive and independent learning rates inspired by adaptive subgradient methods. We apply our approach to solve various linear and non-linear PDEs.

References in corpus (12)

• An overview of gradient descent optimization algorithms
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• DGM: A deep learning algorithm for solving partial differential equations
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain
• Visualizing the Loss Landscape of Neural Nets
• Physics-informed neural networks for solving Reynolds-averaged Navier$\unicode{x2013}$Stokes equations
• Deep Learning of Vortex Induced Vibrations
• Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
• A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture
• Physics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-fidelity Data Fusion
• Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)

Cited by in corpus (7)

• Residual-based attention in physics-informed neural networks
• Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)
• Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks
• k-space Physics-informed Neural Network (k-PINN) for Compressed Spectral Mapping and Efficient Inversion of Vibrations in Thin Composite Laminates
• Splitting physics-informed neural networks for inferring the dynamics of integer- and fractional-order neuron models
• Constrained or Unconstrained? Neural-Network-Based Equation Discovery from Data
• A physics-informed neural network approach to the point defect model for electrochemical oxide film growth