Tangent Space Separability in Feedforward Neural Networks
arXiv:1912.09306
Abstract
Hierarchical neural networks are exponentially more efficient than their corresponding "shallow" counterpart with the same expressive power, but involve huge number of parameters and require tedious amounts of training. By approximating the tangent subspace, we suggest a sparse representation that enables switching to shallow networks, GradNet after a very early training stage. Our experiments show that the proposed approximation of the metric improves and sometimes even surpasses the achievable performance of the original network significantly even after a few epochs of training the original feedforward network.
arXiv comment10 pages; accepted at Workshop "Beyond First-Order Optimization Methods in Machine Learning", 33rd Conference on Neural Information Processing Systems (NeurIPS 2019). arXiv admin note: substantial text overlap with arXiv:1807.06630
References in corpus (9)
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- The Loss Surfaces of Multilayer Networks
- DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Sharp Minima Can Generalize For Deep Nets
- The power of deeper networks for expressing natural functions
- Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds