Big Neural Networks Waste Capacity

Abstract

This article exposes the failure of some big neural networks to leverage added capacity to reduce underfitting. Past research suggest diminishing returns when increasing the size of neural networks. Our experiments on ImageNet LSVRC-2010 show that this may be due to the fact there are highly diminishing returns for capacity in terms of training error, leading to underfitting. This suggests that the optimization method - first order gradient descent - fails at this regime. Directly attacking this problem, either through the optimization method or the choices of parametrization, may allow to improve the generalization error on large datasets, for which a large capacity is required.

References in corpus (1)

• Improving neural networks by preventing co-adaptation of feature detectors

Cited by in corpus (18)

• Do Deep Nets Really Need to be Deep?
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Knowledge Matters: Importance of Prior Information for Optimization
• Deep Learning of Representations: Looking Forward
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Data augmentation instead of explicit regularization
• Training Neural Networks with Stochastic Hessian-Free Optimization
• The SP theory of intelligence: distinctive features and advantages
• Pruning via Iterative Ranking of Sensitivity Statistics
• Probing the Purview of Neural Networks via Gradient Analysis
• Blending LSTMs into CNNs
• Trainless Model Performance Estimation for Neural Architecture Search
• Multilayer Perceptron Algebra
• SECS: Efficient Deep Stream Processing via Class Skew Dichotomy
• CAZSL: Zero-Shot Regression for Pushing Models by Generalizing Through Context
• Quantization Loss Re-Learning Method
• HALO: Learning to Prune Neural Networks with Shrinkage