REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs

Abstract

Physics-Informed Neural Networks (PINNs) offer a promising approach to simulating physical systems. Still, their application is limited by optimization challenges, mainly due to the lack of activation functions that generalize well across several physical systems. Existing activation functions often lack such flexibility and generalization power. To address this issue, we introduce Rational Exponential Activation (REAct), a generalized form of tanh consisting of four learnable shape parameters. Experiments show that REAct outperforms many standard and benchmark activations, achieving an MSE three orders of magnitude lower than tanh on heat problems and generalizing well to finer grids and points beyond the training domain. It also excels at function approximation tasks and improves noise rejection in inverse problems, leading to more accurate parameter estimates across varying noise levels.

5 pages, 5 tables, 1 figure; Accepted at ICASSP 2025

References in corpus (8)

• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Physics-informed neural networks for inverse problems in nano-optics and metamaterials
• Deep Learning of Vortex Induced Vibrations
• Dying ReLU and Initialization: Theory and Numerical Examples
• Collapse of Deep and Narrow Neural Nets
• Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks
• Learning in Sinusoidal Spaces with Physics-Informed Neural Networks
• Learning Specialized Activation Functions for Physics-informed Neural Networks