Revise Saturated Activation Functions

Bing Xu, Ruitong Huang, Mu Li

In this paper, we revise two commonly used saturated functions, the logistic sigmoid and the hyperbolic tangent (tanh). We point out that, besides the well-known non-zero centered…