Residual-based attention in physics-informed neural networks

Abstract

Driven by the need for more efficient and seamless integration of physical models and data, physics-informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains a challenge. In this work, we propose an efficient, gradient-less weighting scheme for PINNs that accelerates the convergence of dynamic or static systems. This simple yet effective attention mechanism is a bounded function of the evolving cumulative residuals and aims to make the optimizer aware of problematic regions at no extra computational cost or adversarial learning. We illustrate that this general method consistently achieves one order of magnitude faster convergence than vanilla PINNs and a minimum relative $L^2$ error of $\mathcal{O}(10^{-5})$, on typical benchmarks of the literature. The method is further tested on the inverse solution of the Navier-Stokes within the brain perivascular spaces, where it considerably improves the prediction accuracy. Furthermore, an ablation study is performed for each case to identify the contribution of the components that enhance the vanilla PINN formulation. Evident from the convergence trajectories is the ability of the optimizer to effectively escape from poor local minima or saddle points while focusing on the challenging domain regions, which consistently have a high residual score. We believe that alongside exact boundary conditions and other model reparameterizations, this type of attention mask could be an essential element for fast training of both PINNs and neural operators.

References in corpus (5)

• A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
• Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)
• Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies
• A Dimension-Augmented Physics-Informed Neural Network (DaPINN) with High Level Accuracy and Efficiency
• Artificial to Spiking Neural Networks Conversion for Scientific Machine Learning

Cited by in corpus (16)

• Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?
• Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks
• Residual-based Attention Physics-informed Neural Networks for Spatio-Temporal Ageing Assessment of Transformers Operated in Renewable Power Plants
• Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks
• Learning in PINNs: Phase transition, diffusion equilibrium, and generalization
• Physics-informed Deep Learning to Solve Three-dimensional Terzaghi Consolidation Equation: Forward and Inverse Problems
• Are Two Hidden Layers Still Enough for the Physics-Informed Neural Networks?
• Training Deep Physics-Informed Kolmogorov-Arnold Networks
• An approximate Riemann solver approach in Physics-Informed Neural Networks for hyperbolic conservation laws
• Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications
• Disentangling Aleatoric and Epistemic Uncertainty in Physics-Informed Neural Networks. Application to Insulation Material Degradation Prognostics
• Quantum Physics-Informed Neural Networks for Maxwell's Equations: Circuit Design, "Black Hole" Barren Plateaus Mitigation, and GPU Acceleration
• Discontinuity-aware KAN-based physics-informed neural networks
• Turbulence Closure in RANS and Flow Inference around a Cylinder using PINNs and Sparse Experimental Data
• Process-Informed Forecasting of Complex Thermal Dynamics in Pharmaceutical Manufacturing
• Stability in Training PINNs for Stiff PDEs: Why Initial Conditions Matter