Expressive power of outer product manifolds on feed-forward neural networks
arXiv:1807.06630
Abstract
Hierarchical neural networks are exponentially more efficient than their corresponding "shallow" counterpart with the same expressive power, but involve huge number of parameters and require tedious amounts of training. Our main idea is to mathematically understand and describe the hierarchical structure of feedforward neural networks by reparametrization invariant Riemannian metrics. By computing or approximating the tangent subspace, we better utilize the original network via sparse representations that enables switching to shallow networks after a very early training stage. Our experiments show that the proposed approximation of the metric improves and sometimes even surpasses the achievable performance of the original network significantly even after a few epochs of training the original feedforward network.
arXiv comment11 pages, 8 figures, under submission
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Overcoming catastrophic forgetting in neural networks
- ADADELTA: An Adaptive Learning Rate Method
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Understanding deep learning requires rethinking generalization
- The Loss Surfaces of Multilayer Networks
- Predicting Parameters in Deep Learning
- The power of deeper networks for expressing natural functions