FeTa: A DCA Pruning Algorithm with Generalization Error Guarantees
arXiv:1803.04239
Abstract
Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classification accuracy. We start by proposing a cheap pruning algorithm for fully connected DNN layers based on difference of convex functions (DC) optimisation, that requires little or no retraining. We then provide a theoretical analysis for the growth in the Generalization Error (GE) of a DNN for the case of bounded perturbations to the hidden layers, of which weight pruning is a special case. Our pruning method is orders of magnitude faster than competing approaches, while our theoretical analysis sheds light to previously observed problems in DNN pruning. Experiments on commnon feedforward neural networks validate our results.
References in corpus (13)
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Weight Uncertainty in Neural Networks
- Tensorizing Neural Networks
- Compressing Neural Networks with the Hashing Trick
- Bayesian Compression for Deep Learning
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Robust Large Margin Deep Neural Networks
- Spectrally-normalized margin bounds for neural networks
- Loss-aware Binarization of Deep Networks
- On the Expressive Power of Deep Neural Networks
- Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
- Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee
- The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning