Zorro: A Flexible and Differentiable Parametric Family of Activation Functions That Extends ReLU and GELU

Matias Roodschild, Jorge Gotay-Sardiñas, Victor A. Jimenez +1

Even in recent neural network architectures such as Transformers and Extended LSTM (xLSTM), and traditional ones like Convolutional Neural Networks, Activation Functions are an int…