Fast ConvNets Using Group-wise Brain Damage
arXiv:1506.02515
Abstract
We revisit the idea of brain damage, i.e. the pruning of the coefficients of a neural network, and suggest how brain damage can be modified and used to speedup convolutional layers. The approach uses the fact that many efficient implementations reduce generalized convolutions to matrix multiplications. The suggested brain damage process prunes the convolutional kernel tensor in a group-wise fashion by adding group-sparsity regularization to the standard training process. After such group-wise pruning, convolutions can be reduced to multiplications of thinned dense matrices, which leads to speedup. In the comparison on AlexNet, the method achieves very competitive performance.
References in corpus (9)
- Distilling the Knowledge in a Neural Network
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Deep Learning with Limited Numerical Precision
- cuDNN: Efficient Primitives for Deep Learning
- MatConvNet - Convolutional Neural Networks for MATLAB
- Tensorizing Neural Networks
- Compressing Neural Networks with the Hashing Trick
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Memory Bounded Deep Convolutional Networks
Cited by in corpus (10)
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Channel Pruning for Accelerating Very Deep Neural Networks
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning
- ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
- Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
- Compact Deep Convolutional Neural Networks With Coarse Pruning
- Quantized Convolutional Neural Networks for Mobile Devices
- Training Sparse Neural Networks using Compressed Sensing
- Jointly Sparse Convolutional Neural Networks in Dual Spatial-Winograd Domains
- A scalable convolutional neural network for task-specified scenarios via knowledge distillation