Multi-Objective Loss Balancing for Physics-Informed Deep Learning

Abstract

Physics-Informed Neural Networks (PINN) are algorithms from deep learning leveraging physical laws by including partial differential equations together with a respective set of boundary and initial conditions as penalty terms into their loss function. In this work, we observe the significant role of correctly weighting the combination of multiple competitive loss functions for training PINNs effectively. To this end, we implement and evaluate different methods aiming at balancing the contributions of multiple terms of the PINNs loss function and their gradients. After reviewing of three existing loss scaling approaches (Learning Rate Annealing, GradNorm and SoftAdapt), we propose a novel self-adaptive loss balancing scheme for PINNs named \emph{ReLoBRaLo} (Relative Loss Balancing with Random Lookback). We extensively evaluate the performance of the aforementioned balancing schemes by solving both forward as well as inverse problems on three benchmark PDEs for PINNs: Burgers' equation, Kirchhoff's plate bending equation and Helmholtz's equation. The results show that ReLoBRaLo is able to consistently outperform the baseline of existing scaling methods in terms of accuracy, while also inducing significantly less computational overhead.

References in corpus (12)

• A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
• Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS
• A Deep Collocation Method for the Bending Analysis of Kirchhoff Plate
• A Physics Informed Neural Network for Time-Dependent Nonlinear and Higher Order Partial Differential Equations
• A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
• A novel meta-learning initialization method for physics-informed neural networks
• SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions
• Theory of Deep Learning IIb: Optimization Properties of SGD
• Accelerating Physics-Informed Neural Network Training with Prior Dictionaries
• PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs

Cited by in corpus (10)

• Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities
• Physics-informed Deep Learning to Solve Three-dimensional Terzaghi Consolidation Equation: Forward and Inverse Problems
• PINN-Obs: Physics-Informed Neural Network-Based Observer for Nonlinear Dynamical Systems
• Physics-Informed Neural Networks for Nonlocal Beam Eigenvalue Problems
• Variational PINNs with tree-based integration and boundary element data in the modeling of multi-phase architected materials
• Bayesian Reasoning for Physics Informed Neural Networks
• The magnetic scalar potential and demagnetization vector for a cylinder tile
• Hamiltonian-reconstruction distance as a success metric for the Variational Quantum Eigensolver
• Higher-Order LaSDI: Reduced Order Modeling with Multiple Time Derivatives
• A Variational Kolosov--Muskhelishvili Network for Elasticity and Fracture