Structured Pruning of Deep Convolutional Neural Networks

Abstract

Real time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also do not fit well on parallel computation. We introduce structured sparsity at various scales for convolutional neural networks, which are channel wise, kernel wise and intra kernel strided sparsity. This structured sparsity is very advantageous for direct computational resource savings on embedded computers, parallel computing environments and hardware based systems. To decide the importance of network connections and paths, the proposed method uses a particle filtering approach. The importance weight of each particle is assigned by computing the misclassification rate with corresponding connectivity pattern. The pruned network is re-trained to compensate for the losses due to pruning. While implementing convolutions as matrix products, we particularly show that intra kernel strided sparsity with a simple constraint can significantly reduce the size of kernel and feature map matrices. The pruned network is finally fixed point optimized with reduced word length precision. This results in significant reduction in the total storage size providing advantages for on-chip memory based implementations of deep neural networks.

arXiv comment11 pages, 8 figures, 1 table

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• cuDNN: Efficient Primitives for Deep Learning
• Resiliency of Deep Neural Networks under Quantization
• Memory Bounded Deep Convolutional Networks

Cited by in corpus (23)

• Pruning Convolutional Neural Networks for Resource Efficient Inference
• A Review on Deep Learning Techniques Applied to Semantic Segmentation
• Pruning Filters for Efficient ConvNets
• Channel Pruning for Accelerating Very Deep Neural Networks
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• Task dependent Deep LDA pruning of neural networks
• MPNA: A Massively-Parallel Neural Array Accelerator with Dataflow Optimization for Convolutional Neural Networks
• An Experimental Study of the Impact of Pre-training on the Pruning of a Convolutional Neural Network
• Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning
• Structured Deep Neural Network Pruning via Matrix Pivoting
• Deleting object selective units in a fully-connected layer of deep convolutional networks improves classification performance
• Baseline Pruning-Based Approach to Trojan Detection in Neural Networks
• MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• Quantized neural network design under weight capacity constraint
• NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse Networks
• Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip
• Towards thinner convolutional neural networks through Gradually Global Pruning
• Prune the Convolutional Neural Networks with Sparse Shrink
• Grow-Push-Prune: aligning deep discriminants for effective structural network compression
• Pay Attention to Convolution Filters: Towards Fast and Accurate Fine-Grained Transfer Learning