Meta-learning PINN loss functions

Abstract

We propose a meta-learning technique for offline discovery of physics-informed neural network (PINN) loss functions. We extend earlier works on meta-learning, and develop a gradient-based meta-learning algorithm for addressing diverse task distributions based on parametrized partial differential equations (PDEs) that are solved with PINNs. Furthermore, based on new theory we identify two desirable properties of meta-learned losses in PINN problems, which we enforce by proposing a new regularization method or using a specific parametrization of the loss function. In the computational examples, the meta-learned losses are employed at test time for addressing regression and PDE task distributions. Our results indicate that significant performance improvement can be achieved by using a shared-among-tasks offline-learned loss function even for out-of-distribution meta-testing. In this case, we solve for test tasks that do not belong to the task distribution used in meta-training, and we also employ PINN architectures that are different from the PINN architecture used in meta-training. To better understand the capabilities and limitations of the proposed method, we consider various parametrizations of the loss function and describe different algorithm design options and how they may affect meta-learning performance.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• DeepXDE: A deep learning library for solving differential equations
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition
• A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems
• On First-Order Meta-Learning Algorithms
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Meta-Gradient Reinforcement Learning
• Evolved Policy Gradients
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Generalized Inner Loop Meta-Learning
• Learning to Teach with Dynamic Loss Functions
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Reward Shaping via Meta-Learning
• On Learning Intrinsic Rewards for Policy Gradient Methods
• Learning Surrogate Losses
• Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment
• Learning to learn via Self-Critique
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• Effective Regularization Through Loss-Function Metalearning

Cited by in corpus (14)

• A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
• Super-Resolution Analysis via Machine Learning: A Survey for Fluid Flows
• Physics-informed neural network methods based on Miura transformations and discovery of new localized wave solutions
• Physics-Informed Deep Learning For Traffic State Estimation: A Survey and the Outlook
• HANNA: Hard-constraint Neural Network for Consistent Activity Coefficient Prediction
• Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss
• Accelerating physics-informed neural network based 1D arc simulation by meta learning
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities
• Meta-Auto-Decoder: A Meta-Learning Based Reduced Order Model for Solving Parametric Partial Differential Equations
• Neural Networks with Inputs Based on Domain of Dependence and A Converging Sequence for Solving Conservation Laws, Part I: 1D Riemann Problems
• Gated X-TFC: Soft Domain Decomposition for Forward and Inverse Problems in Sharp-Gradient PDEs