Multi-Objective Loss Balancing for Physics-Informed Deep Learning

Abstract

Physics-Informed Neural Networks (PINN) are algorithms from deep learning leveraging physical laws by including partial differential equations together with a respective set of boundary and initial conditions as penalty terms into their loss function. In this work, we observe the significant role of correctly weighting the combination of multiple competitive loss functions for training PINNs effectively. To this end, we implement and evaluate different methods aiming at balancing the contributions of multiple terms of the PINNs loss function and their gradients. After reviewing of three existing loss scaling approaches (Learning Rate Annealing, GradNorm and SoftAdapt), we propose a novel self-adaptive loss balancing scheme for PINNs named \emph{ReLoBRaLo} (Relative Loss Balancing with Random Lookback). We extensively evaluate the performance of the aforementioned balancing schemes by solving both forward as well as inverse problems on three benchmark PDEs for PINNs: Burgers' equation, Kirchhoff's plate bending equation and Helmholtz's equation. The results show that ReLoBRaLo is able to consistently outperform the baseline of existing scaling methods in terms of accuracy, while also inducing significantly less computational overhead.

References in corpus (25)

• Adam: A Method for Stochastic Optimization
• DeepXDE: A deep learning library for solving differential equations
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• DGM: A deep learning algorithm for solving partial differential equations
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition
• A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
• PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs
• Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS
• A Deep Collocation Method for the Bending Analysis of Kirchhoff Plate
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations
• Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations
• A Physics Informed Neural Network for Time-Dependent Nonlinear and Higher Order Partial Differential Equations
• On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs
• Transfer learning based multi-fidelity physics informed deep neural network
• Geometric deep learning for computational mechanics Part I: Anisotropic Hyperelasticity
• A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
• A novel meta-learning initialization method for physics-informed neural networks
• SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions
• Theory of Deep Learning IIb: Optimization Properties of SGD
• Accelerating Physics-Informed Neural Network Training with Prior Dictionaries
• PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs

Cited by in corpus (10)

• Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities
• Physics-informed Deep Learning to Solve Three-dimensional Terzaghi Consolidation Equation: Forward and Inverse Problems
• PINN-Obs: Physics-Informed Neural Network-Based Observer for Nonlinear Dynamical Systems
• Physics-Informed Neural Networks for Nonlocal Beam Eigenvalue Problems
• Variational PINNs with tree-based integration and boundary element data in the modeling of multi-phase architected materials
• Bayesian Reasoning for Physics Informed Neural Networks
• The magnetic scalar potential and demagnetization vector for a cylinder tile
• Hamiltonian-reconstruction distance as a success metric for the Variational Quantum Eigensolver
• Higher-Order LaSDI: Reduced Order Modeling with Multiple Time Derivatives
• A Variational Kolosov--Muskhelishvili Network for Elasticity and Fracture