On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

Abstract

Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction-diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at \url{https://github.com/PredictiveIntelligenceLab/MultiscalePINNs}.

arXiv comment27 pages, 18 figures

References in corpus (32)

• Adam: A Method for Stochastic Optimization
• Array Programming with NumPy
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• Solving high-dimensional partial differential equations using deep learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data
• Fourier Neural Operator for Parametric Partial Differential Equations
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Physics-informed neural networks for inverse problems in nano-optics and metamaterials
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• Simulator-free Solution of High-Dimensional Stochastic Elliptic Partial Differential Equations using Deep Neural Networks
• On the Spectral Bias of Neural Networks
• When and why PINNs fail to train: A neural tangent kernel perspective
• Deep learning of free boundary and Stefan problems
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Learning Unknown Physics of non-Newtonian Fluids
• Towards Understanding the Spectral Bias of Deep Learning
• Multi-scale Deep Neural Network (MscaleDNN) Methods for Oscillatory Stokes Flows in Complex Domains
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations
• Gradient Dynamics of Shallow Univariate ReLU Networks
• Error estimates of residual minimization using neural networks for linear PDEs
• Bayesian differential programming for robust systems identification under uncertainty

Cited by in corpus (81)

• Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
• CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarios
• Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov Arnold Networks
• Residual-based attention in physics-informed neural networks
• Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training
• A physics-informed neural network for quantifying the microstructure properties of polycrystalline Nickel using ultrasound data
• Characterizing possible failure modes in physics-informed neural networks
• Physics-informed neural networks of the Saint-Venant equations for downscaling a large-scale river model
• A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions
• Mixed formulation of physics-informed neural networks for thermo-mechanically coupled systems and heterogeneous domains
• Multilevel domain decomposition-based architectures for physics-informed neural networks
• SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs
• Learning in Sinusoidal Spaces with Physics-Informed Neural Networks
• CENN: Conservative energy method based on neural networks with subdomains for solving variational problems involving heterogeneous and complex geometries
• Physics-informed radial basis network (PIRBN): A local approximating neural network for solving nonlinear PDEs
• An overview on deep learning-based approximation methods for partial differential equations
• Uncovering turbulent plasma dynamics via deep learning from partial observations
• A practical PINN framework for multi-scale problems with multi-magnitude loss terms
• PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting
• Physics-informed neural network solution of thermo-hydro-mechanical (THM) processes in porous media
• Enhanced physics-informed neural networks for hyperelasticity
• Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?
• Pre-training strategy for solving evolution equations based on physics-informed neural networks
• BINN: A deep learning approach for computational mechanics problems based on boundary integral equations
• Physics-informed Neural Network Estimation of Material Properties in Soft Tissue Nonlinear Biomechanical Models
• Deep energy method in topology optimization applications
• $\Delta$-PINNs: physics-informed neural networks on complex geometries
• Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems
• Efficient physics-informed neural networks using hash encoding
• DCEM: A deep complementary energy method for solid mechanics
• Bayesian Physics Informed Neural Networks for Data Assimilation and Spatio-Temporal Modelling of Wildfires
• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors
• An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operator
• Aero-Nef: Neural Fields for Rapid Aircraft Aerodynamics Simulations
• WarpPINN: Cine-MR image registration with physics-informed neural networks
• Improved Training of Physics-Informed Neural Networks with Model Ensembles
• MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs
• HomPINNs: homotopy physics-informed neural networks for solving the inverse problems of nonlinear differential equations with multiple solutions
• CEENs: Causality-enforced evolutional networks for solving time-dependent partial differential equations
• Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks
• A Mathematical Guide to Operator Learning
• Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks
• Solving and Learning Nonlinear PDEs with Gaussian Processes
• I-FENN for thermoelasticity based on physics-informed temporal convolutional network (PI-TCN)
• k-space Physics-informed Neural Network (k-PINN) for Compressed Spectral Mapping and Efficient Inversion of Vibrations in Thin Composite Laminates
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• An extrapolation-driven network architecture for physics-informed deep learning
• Adapting Physics-Informed Neural Networks to Improve ODE Optimization in Mosquito Population Dynamics
• Machine Learning in Acoustics: A Review and Open-Source Repository
• Physics Informed Neural Networks for Modeling of 3D Flow-Thermal Problems with Sparse Domain Data
• A Priori Analysis of Stable Neural Network Solutions to Numerical PDEs
• Learning in PINNs: Phase transition, diffusion equilibrium, and generalization
• Multiscale Physics-Informed Neural Networks for the Inverse Design of Hyperuniform Optical Materials
• LSA-PINN: Linear Boundary Connectivity Loss for Solving PDEs on Complex Geometry
• Simulation and Prediction of Countercurrent Spontaneous Imbibition at Early and Late Times Using Physics-Informed Neural Networks
• Interpretable structural model error discovery from sparse assimilation increments using spectral bias-reduced neural networks: A quasi-geostrophic turbulence test case
• Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities
• Physics-informed Deep Learning to Solve Three-dimensional Terzaghi Consolidation Equation: Forward and Inverse Problems
• Automated differential equation solver based on the parametric approximation optimization
• Physics-Informed Neural Networks for Transonic Flows around an Airfoil
• Training Deep Physics-Informed Kolmogorov-Arnold Networks
• Physics-Informed Graph-Mesh Networks for PDEs: A hybrid approach for complex problems
• Multi-variance replica exchange stochastic gradient MCMC for inverse and forward Bayesian physics-informed neural network
• Fast PDE-constrained optimization via self-supervised operator learning
• Time-dependent Neural Galerkin Method for Quantum Dynamics
• Sequential learning based PINNs to overcome temporal domain complexities in unsteady flow past flapping wings
• Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks
• A deep neural network approach on solving the linear transport model under diffusive scaling
• Eig-PIELM: A Mesh-Free Approach for Efficient Eigen-Analysis with Physics-Informed Extreme Learning Machines
• An approximate Riemann solver approach in Physics-Informed Neural Networks for hyperbolic conservation laws
• Physics-Informed Neural Network for Elastic Wave-Mode Separation
• Discontinuity-aware KAN-based physics-informed neural networks
• Quantum Physics-Informed Neural Networks for Maxwell's Equations: Circuit Design, "Black Hole" Barren Plateaus Mitigation, and GPU Acceleration
• Physics-informed neural networks to learn cardiac fiber orientation from multiple electroanatomical maps
• A Novel Fourier Feature Network for Solving Partial Differential Equations
• How Sparse and How Noisy? Systematic Benchmarking of Inverse Physics-Informed Neural Networks for Manning Friction Estimation in Shallow Water Equations
• Operator learning for models of tear film breakup
• Transforming physics-informed machine learning to convex optimization
• A Neural Network for the Identical Kuramoto Equation: Architectural Considerations and Performance Evaluation