Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks
arXiv:2407.01613 · doi:10.1016/j.jcp.2025.114226
Abstract
Physics-informed deep learning has emerged as a promising alternative for solving partial differential equations. However, for complex problems, training these networks can still be challenging, often resulting in unsatisfactory accuracy and efficiency. In this work, we demonstrate that the failure of plain physics-informed neural networks arises from the significant discrepancy in the convergence rate of residuals at different training points, where the slowest convergence rate dominates the overall solution convergence. Based on these observations, we propose a pointwise adaptive weighting method that balances the residual decay rate across different training points. The performance of our proposed adaptive weighting method is compared with current state-of-the-art adaptive weighting methods on benchmark problems for both physics-informed neural networks and physics-informed deep operator networks. Through extensive numerical results we demonstrate that our proposed approach of balanced residual decay rates offers several advantages, including bounded weights, high prediction accuracy, fast convergence rate, low training uncertainty, low computational cost, and ease of hyperparameter tuning.
13 figures, 4 tables
References in corpus (16)
- DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
- A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems
- On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
- A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
- A physics-informed variational DeepONet for predicting the crack path in brittle materials
- A Physics Informed Neural Network for Time-Dependent Nonlinear and Higher Order Partial Differential Equations
- Residual-based attention in physics-informed neural networks
- A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture
- Discontinuity Computing using Physics-Informed Neural Network
- Physics and Equality Constrained Artificial Neural Networks: Application to Forward and Inverse Problems with Multi-fidelity Data Fusion
- Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems
- Investigating and Mitigating Failure Modes in Physics-informed Neural Networks (PINNs)
- A multifidelity approach to continual learning for physical systems