Adaptive activation functions accelerate convergence in deep and physics-informed neural networks

Abstract

We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems.

24 pages, 21 figures

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• An overview of gradient descent optimization algorithms
• Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
• Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
• Machine Learning of Linear Differential Equations using Gaussian Processes
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Deep Learning of Vortex Induced Vibrations
• A Closer Look at Memorization in Deep Networks
• Inferring solutions of differential equations using noisy multi-fidelity data
• Data-driven discovery of PDEs in complex datasets
• On the Spectral Bias of Neural Networks

Cited by in corpus (132)

• DeepXDE: A deep learning library for solving differential equations
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition
• A Physics-Informed Machine Learning Approach for Solving Heat Transfer Equation in Advanced Manufacturing and Engineering Applications
• Physics-informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions
• Deep Autoencoder based Energy Method for the Bending, Vibration, and Buckling Analysis of Kirchhoff Plates
• Efficient training of physics-informed neural networks via importance sampling
• Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks
• Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarios
• Variational Physics-Informed Neural Networks For Solving Partial Differential Equations
• Simulation of multi-species flow and heat transfer using physics-informed neural networks
• Residual-based attention in physics-informed neural networks
• A two-stage physics-informed neural network method based on conserved quantities and applications in localized wave solutions
• Multi-Objective Loss Balancing for Physics-Informed Deep Learning
• Physics Informed Neural Networks for Simulating Radiative Transfer
• Deep Learning in Deterministic Computational Mechanics
• Theory-guided hard constraint projection (HCP): a knowledge-based data-driven scientific machine learning method
• Physics-informed neural networks for the shallow-water equations on the sphere
• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• Meta-learning PINN loss functions
• Hybrid FEM-NN models: Combining artificial neural networks with the finite element method
• A physics-informed neural network for quantifying the microstructure properties of polycrystalline Nickel using ultrasound data
• Distributed physics informed neural network for data-efficient solution to partial differential equations
• A Gradient-based Deep Neural Network Model for Simulating Multiphase Flow in Porous Media
• Physics-Informed Neural Networks for Shell Structures
• A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions
• CENN: Conservative energy method based on neural networks with subdomains for solving variational problems involving heterogeneous and complex geometries
• A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems
• Data-driven vector localized waves and parameters discovery for Manakov system using deep learning approach
• An overview on deep learning-based approximation methods for partial differential equations
• Learning hidden elasticity with deep neural networks
• Physics-informed neural network solution of thermo-hydro-mechanical (THM) processes in porous media
• A novel meta-learning initialization method for physics-informed neural networks
• Reinforcement Learning of Control Strategies for Reducing Skin Friction Drag in a Fully Developed Channel Flow
• Magnetostatics and micromagnetics with physics informed neural networks
• Physics-Informed Neural Networks for Nonhomogeneous Material Identification in Elasticity Imaging
• Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?
• Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics
• Theory of the Frequency Principle for General Deep Neural Networks
• Weak Form Theory-guided Neural Network (TgNN-wf) for Deep Learning of Subsurface Single and Two-phase Flow
• Enriched physics-informed neural networks for in-plane crack problems: Theory and MATLAB codes
• BINN: A deep learning approach for computational mechanics problems based on boundary integral equations
• Physics-Informed Deep Learning For Traffic State Estimation: A Survey and the Outlook
• Physics-informed neural networks method in high-dimensional integrable systems
• Algorithmically-Consistent Deep Learning Frameworks for Structural Topology Optimization
• Universal Activation Function For Machine Learning
• $\Delta$-PINNs: physics-informed neural networks on complex geometries
• Data-driven discovery of Green's functions with human-understandable deep learning
• Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domains
• Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks
• Optimizing a DIscrete Loss (ODIL) to solve forward and inverse problems for partial differential equations using machine learning tools
• Low-temperature plasma simulation based on physics-informed neural networks: frameworks and preliminary applications
• An extended physics informed neural network for preliminary analysis of parametric optimal control problems
• Recipes for when Physics Fails: Recovering Robust Learning of Physics Informed Neural Networks
• Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem
• An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operator
• Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks
• RiemannONets: Interpretable Neural Operators for Riemann Problems
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• Physics-Informed Neural Networks for Parametric Compressible Euler Equations
• On the Use of Neural Networks for Full Waveform Inversion
• ModalPINN: an extension of Physics-Informed Neural Networks with enforced truncated Fourier decomposition for periodic flow reconstruction using a limited number of imperfect sensors
• Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• Neural Eikonal Solver: improving accuracy of physics-informed neural networks for solving eikonal equation in case of caustics
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies
• Physics-informed neural networks for inverse problems in supersonic flows
• Can Transfer Neuroevolution Tractably Solve Your Differential Equations?
• CEENs: Causality-enforced evolutional networks for solving time-dependent partial differential equations
• Accelerating Physics-Informed Neural Network Training with Prior Dictionaries
• Simple and complex spiking neurons: perspectives and analysis in a simple STDP scenario
• Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks
• Stochastic particle advection velocimetry (SPAV): theory, simulations, and proof-of-concept experiments
• Physics-informed neural networks for unsteady incompressible flows with time-dependent moving boundaries
• A Dimension-Augmented Physics-Informed Neural Network (DaPINN) with High Level Accuracy and Efficiency
• Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks
• Opening the Black Box: Towards inherently interpretable energy data imputation models using building physics insight
• Constrained Hamiltonian Systems and Physics-Informed Neural Networks: Hamilton-Dirac Neural Networks
• k-space Physics-informed Neural Network (k-PINN) for Compressed Spectral Mapping and Efficient Inversion of Vibrations in Thin Composite Laminates
• Neural networks: solving the chemistry of the interstellar medium
• Improved Surrogate Modeling of Fluid Dynamics with Physics-Informed Neural Networks
• SVD-PINNs: Transfer Learning of Physics-Informed Neural Networks via Singular Value Decomposition
• Adapting Physics-Informed Neural Networks to Improve ODE Optimization in Mosquito Population Dynamics
• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• Multiple scattering simulation via physics-informed neural networks
• Solving the Pulsar Equation using Physics-Informed Neural Networks
• Certified machine learning: A posteriori error estimation for physics-informed neural networks
• Learning in PINNs: Phase transition, diffusion equilibrium, and generalization
• Fractional and tempered fractional models for Reynolds-averaged Navier-Stokes equations
• Geometry-aware framework for deep energy method: an application to structural mechanics with hyperelastic materials
• Reproducing Activation Function for Deep Learning
• A Continuous Convolutional Trainable Filter for Modelling Unstructured Data
• Splitting physics-informed neural networks for inferring the dynamics of integer- and fractional-order neuron models
• Learning of viscosity functions in rarefied gas flows with physics-informed neural networks
• Discovering Physics-Informed Neural Networks Model for Solving Partial Differential Equations through Evolutionary Computation
• Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities
• Physics-Informed Neural Networks for Transonic Flows around an Airfoil
• Neuronal diversity can improve machine learning for physics and beyond
• Physics Informed Deep Learning for Flow and Transport in Porous Media
• On the convergence of PINNs
• Sequential learning based PINNs to overcome temporal domain complexities in unsteady flow past flapping wings
• Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks
• Physics-regularized neural network of the ideal-MHD solution operator in Wendelstein 7-X configurations
• Deep frequency principle towards understanding why deeper learning is faster
• Adaptive Informed Deep Neural Networks for Power Flow Analysis
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• A Neural Solver for Variational Problems on CAD Geometries with Application to Electric Machine Simulation
• Gradient Flow Based Phase-Field Modeling Using Separable Neural Networks
• Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Control
• Bayesian Hidden Physics Models: Uncertainty Quantification for Discovery of Nonlinear Partial Differential Operators from Data
• Adaptive Physics-Informed Neural Networks for Markov-Chain Monte Carlo
• Deep ReLU networks and high-order finite element methods II: Chebyshev emulation
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• Generalization Error Analysis of Neural networks with Gradient Based Regularization
• Finding Closure Terms Directly from Coarse Data for 2D Turbulent Flow
• Discontinuity-aware KAN-based physics-informed neural networks
• Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications
• A New Artificial Neuron Proposal with Trainable Simultaneous Local and Global Activation Function
• REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• Fractional models of Reynolds-averaged Navier-Stokes equations for Turbulent flows
• Application of mixed-variable physics-informed neural networks to solve normalised momentum and energy transport equations for 2D internal convective flow
• Physics-informed neural networks for non-Newtonian fluid thermo-mechanical problems: an application to rubber calendering process
• Learning To Estimate Regions Of Attraction Of Autonomous Dynamical Systems Using Physics-Informed Neural Networks
• A Variational Kolosov--Muskhelishvili Network for Elasticity and Fracture
• Spatio-Temporal Uncertainty-Modulated Physics-Informed Neural Networks for Solving Hyperbolic Conservation Laws with Strong Shocks
• Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning
• Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating a class of inverse problems for PDEs
• How Sparse and How Noisy? Systematic Benchmarking of Inverse Physics-Informed Neural Networks for Manning Friction Estimation in Shallow Water Equations
• The data-driven localized wave solutions of the derivative nonlinear Schrodinger equation by using improved PINN approach