Training Feedforward Neural Networks with Standard Logistic Activations is Feasible
arXiv:1710.01013
Abstract
Training feedforward neural networks with standard logistic activations is considered difficult because of the intrinsic properties of these sigmoidal functions. This work aims at showing that these networks can be trained to achieve generalization performance comparable to those based on hyperbolic tangent activations. The solution consists on applying a set of conditions in parameter initialization, which have been derived from the study of the properties of a single neuron from an information-theoretic perspective. The proposed initialization is validated through an extensive experimental analysis.
References in corpus (7)
- Deep Learning in Neural Networks: An Overview
- Towards A Rigorous Science of Interpretable Machine Learning
- European Union regulations on algorithmic decision-making and a "right to explanation"
- Exploring the Limits of Language Modeling
- On the Number of Linear Regions of Deep Neural Networks
- The Marginal Value of Adaptive Gradient Methods in Machine Learning
- Random Walk Initialization for Training Very Deep Feedforward Networks