Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions

Eylon E. Krause

The choice of activation function plays a crucial role in the optimization and performance of deep neural networks. While the Rectified Linear Unit (ReLU) remains the dominant choi…