Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks

Abstract

We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each layer (layer-wise) and for every neuron (neuron-wise) separately, and then optimizing it using a variant of stochastic gradient descent algorithm. In order to further increase the training speed, an activation slope based slope recovery term is added in the loss function, which further accelerates convergence, thereby reducing the training cost. On the theoretical side, we prove that in the proposed method, the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate, and that the gradient dynamics of the proposed method is not achievable by base methods with any (adaptive) learning rates. We further show that the adaptive activation methods accelerate the convergence by implicitly multiplying conditioning matrices to the gradient of the base method without any explicit computation of the conditioning matrix and the matrix-vector product. The different adaptive activation functions are shown to induce different implicit conditioning matrices. Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.

arXiv comment19 pages, 8 figures

References in corpus (2)

• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Solving inverse-PDE problems with physics-aware neural networks

Cited by in corpus (24)

• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• Physics-informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions
• PINNeik: Eikonal solution using physics-informed neural networks
• A two-stage physics-informed neural network method based on conserved quantities and applications in localized wave solutions
• A physics-informed neural network for quantifying the microstructure properties of polycrystalline Nickel using ultrasound data
• CENN: Conservative energy method based on neural networks with subdomains for solving variational problems involving heterogeneous and complex geometries
• Data-driven vector localized waves and parameters discovery for Manakov system using deep learning approach
• Universal Activation Function For Machine Learning
• Physical Activation Functions (PAFs): An Approach for More Efficient Induction of Physics into Physics-Informed Neural Networks (PINNs)
• An extended physics informed neural network for preliminary analysis of parametric optimal control problems
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• Physics-Informed Neural Networks for Parametric Compressible Euler Equations
• Neural Eikonal Solver: improving accuracy of physics-informed neural networks for solving eikonal equation in case of caustics
• Predicting nonlinear dynamics of optical solitons in optical fiber via the SCPINN
• Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies
• Physics-informed neural networks for inverse problems in supersonic flows
• Multispecies bird sound recognition using a fully convolutional neural network
• k-space Physics-informed Neural Network (k-PINN) for Compressed Spectral Mapping and Efficient Inversion of Vibrations in Thin Composite Laminates
• Adapting Physics-Informed Neural Networks to Improve ODE Optimization in Mosquito Population Dynamics
• Modelling parametric uncertainty in PDEs models via Physics-Informed Neural Networks
• A Tutorial on the Use of Physics-Informed Neural Networks to Compute the Spectrum of Quantum Systems
• Physics-Informed Neural Networks for Transonic Flows around an Airfoil
• Finding Closure Terms Directly from Coarse Data for 2D Turbulent Flow
• REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs