Learning in Sinusoidal Spaces with Physics-Informed Neural Networks

Abstract

A physics-informed neural network (PINN) uses physics-augmented loss functions, e.g., incorporating the residual term from governing partial differential equations (PDEs), to ensure its output is consistent with fundamental physics laws. However, it turns out to be difficult to train an accurate PINN model for many problems in practice. In this paper, we present a novel perspective of the merits of learning in sinusoidal spaces with PINNs. By analyzing behavior at model initialization, we first show that a PINN of increasing expressiveness induces an initial bias around flat output functions. Notably, this initial solution can be very close to satisfying many physics PDEs, i.e., falling into a local minimum of the PINN loss that only minimizes PDE residuals, while still being far from the true solution that jointly minimizes PDE residuals and the initial and/or boundary conditions. It is difficult for gradient descent optimization to escape from such a local minimum trap, often causing the training to stall. We then prove that the sinusoidal mapping of inputs, in an architecture we label as sf-PINN, is effective to increase input gradient variability, thus avoiding being trapped in such deceptive local minimum. The level of variability can be effectively modulated to match high-frequency patterns in the problem at hand. A key facet of this paper is the comprehensive empirical study that demonstrates the efficacy of learning in sinusoidal spaces with PINNs for a wide range of forward and inverse modelling problems spanning multiple physics domains.

arXiv comment16 pages, 13 figures

References in corpus (27)

• Adam: A Method for Stochastic Optimization
• DeepXDE: A deep learning library for solving differential equations
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• DGM: A deep learning algorithm for solving partial differential equations
• Theory-guided Data Science: A New Paradigm for Scientific Discovery from Data
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Physics-informed neural networks for inverse problems in nano-optics and metamaterials
• A unified deep artificial neural network approach to partial differential equations in complex geometries
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain
• A Physics-Informed Machine Learning Approach for Solving Heat Transfer Equation in Advanced Manufacturing and Engineering Applications
• Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations
• SciANN: A Keras/Tensorflow wrapper for scientific computations and physics-informed deep learning using artificial neural networks
• Deep Learning of Vortex Induced Vibrations
• CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method
• Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling
• A Physics-Guided Neural Network Framework for Elastic Plates: Comparison of Governing Equations-Based and Energy-Based Approaches
• D3M: A deep domain decomposition method for partial differential equations
• Physics-Driven Regularization of Deep Neural Networks for Enhanced Engineering Design and Analysis
• Can Transfer Neuroevolution Tractably Solve Your Differential Equations?
• Adversarial Multi-task Learning Enhanced Physics-informed Neural Networks for Solving Partial Differential Equations
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• CoPhy-PGNN: Learning Physics-guided Neural Networks with Competing Loss Functions for Solving Eigenvalue Problems

Cited by in corpus (13)

• CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method
• Physics-informed neural networks of the Saint-Venant equations for downscaling a large-scale river model
• Data vs. Physics: The Apparent Pareto Front of Physics-Informed Neural Networks
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• Improved Training of Physics-Informed Neural Networks with Model Ensembles
• Noise-aware Physics-informed Machine Learning for Robust PDE Discovery
• Point Neuron Learning: A New Physics-Informed Neural Network Architecture
• Reconstructing unsteady flows from sparse, noisy measurements with a physics-constrained convolutional neural network
• Decoder Decomposition for the Analysis of the Latent Space of Nonlinear Autoencoders With Wind-Tunnel Experimental Data
• Forward and Inverse Simulation of Pseudo-Two-Dimensional Model of Lithium-Ion Batteries Using Neural Networks
• LSA-PINN: Linear Boundary Connectivity Loss for Solving PDEs on Complex Geometry
• Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities
• REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs