A New Artificial Neuron Proposal with Trainable Simultaneous Local and Global Activation Function

Abstract

The activation function plays a fundamental role in the artificial neural network learning process. However, there is no obvious choice or procedure to determine the best activation function, which depends on the problem. This study proposes a new artificial neuron, named global-local neuron, with a trainable activation function composed of two components, a global and a local. The global component term used here is relative to a mathematical function to describe a general feature present in all problem domain. The local component is a function that can represent a localized behavior, like a transient or a perturbation. This new neuron can define the importance of each activation function component in the learning phase. Depending on the problem, it results in a purely global, or purely local, or a mixed global and local activation function after the training phase. Here, the trigonometric sine function was employed for the global component and the hyperbolic tangent for the local component. The proposed neuron was tested for problems where the target was a purely global function, or purely local function, or a composition of two global and local functions. Two classes of test problems were investigated, regression problems and differential equations solving. The experimental tests demonstrated the Global-Local Neuron network's superior performance, compared with simple neural networks with sine or hyperbolic tangent activation function, and with a hybrid network that combines these two simple neural networks.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems
• Prediction of the strength and timing of sunspot cycle 25 reveal decadal-scale space environmental conditions
• Hamiltonian neural networks for solving equations of motion
• Unsupervised Neural Networks for Quantum Eigenvalue Problems
• Semi-supervised Neural Networks solve an inverse problem for modeling Covid-19 spread
• Unsupervised Learning of Solutions to Differential Equations with Generative Adversarial Networks