Faster Convergence & Generalization in DNNs
arXiv:1807.11414
Abstract
Deep neural networks have gained tremendous popularity in last few years. They have been applied for the task of classification in almost every domain. Despite the success, deep networks can be incredibly slow to train for even moderate sized models on sufficiently large datasets. Additionally, these networks require large amounts of data to be able to generalize. The importance of speeding up convergence, and generalization in deep networks can not be overstated. In this work, we develop an optimization algorithm based on generalized-optimal updates derived from minibatches that lead to faster convergence. Towards the end, we demonstrate on two benchmark datasets that the proposed method achieves two orders of magnitude speed up over traditional back-propagation, and is more robust to noise/over-fitting.
arXiv comment9 pages, 6 figures
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Understanding deep learning requires rethinking generalization
- Opening the Black Box of Deep Neural Networks via Information
- Fast Training of Convolutional Networks through FFTs
- Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
- On the Generalization Error Bounds of Neural Networks under Diversity-Inducing Mutual Angular Regularization