Revise Saturated Activation Functions
arXiv:1602.05980
Abstract
In this paper, we revise two commonly used saturated functions, the logistic sigmoid and the hyperbolic tangent (tanh). We point out that, besides the well-known non-zero centered property, slope of the activation function near the origin is another possible reason making training deep networks with the logistic function difficult to train. We demonstrate that, with proper rescaling, the logistic sigmoid achieves comparable results with tanh. Then following the same argument, we improve tahn by penalizing in the negative part. We show that "penalized tanh" is comparable and even outperforms the state-of-the-art non-saturated functions including ReLU and leaky ReLU on deep convolution neural networks. Our results contradict to the conclusion of previous works that the saturation property causes the slow convergence. It suggests further investigation is necessary to better understand activation functions in deep architectures.
References in corpus (3)
Cited by in corpus (11)
- A survey on modern trainable activation functions
- Long-time predictive modeling of nonlinear dynamical systems using neural networks
- One model Packs Thousands of Items with Recurrent Conditional Query Learning
- PLU: The Piecewise Linear Unit Activation Function
- Empirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks
- Effect of Activation Functions on the Training of Overparametrized Neural Nets
- A New Backpropagation Algorithm without Gradient Descent
- S++: A Fast and Deployable Secure-Computation Framework for Privacy-Preserving Neural Network Training
- Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
- Random Bias Initialization Improves Quantized Training
- Revisiting Initialization of Neural Networks