ULU: A Unified Activation Function

Simin Huo

We propose \textbf{ULU}, a novel non-monotonic, piecewise activation function defined as $\{f(x;{\alpha }_1),x<0;f(x;{\alpha }_2),x>=0\}$, where $f(x;\alpha )=0.5x(tanh(\alpha x)+1),\alpha >0$. ULU treats positiv…