A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

Roberto Rossi

The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent Bernoulli gate. This work gives an alternative interpretation: GELU is the expected…