Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks

Abstract

The success of deep neural networks in many real-world applications is leading to new challenges in building more efficient architectures. One effective way of making networks more efficient is neural network compression. We provide an overview of existing neural network compression methods that can be used to make neural networks more efficient by changing the architecture of the network. First, we introduce a new way to categorize all published compression methods, based on the amount of data and compute needed to make the methods work in practice. These are three 'levels of compression solutions'. Second, we provide a taxonomy of tensor factorization based and probabilistic compression methods. Finally, we perform an extensive evaluation of different compression techniques from the literature for models trained on ImageNet. We show that SVD and probabilistic compression or pruning methods are complementary and give the best results of all the considered methods. We also provide practical ways to combine them.

References in corpus (20)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• cuDNN: Efficient Primitives for Deep Learning
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Pruning Filters for Efficient ConvNets
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Variational Dropout and the Local Reparameterization Trick
• The State of Sparsity in Deep Neural Networks
• Convolutional neural networks with low-rank regularization
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Compressing Neural Networks using the Variational Information Bottleneck
• Ultimate tensorization: compressing convolutional and FC layers alike
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Rate Distortion For Model Compression: From Theory To Practice
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Automatic Rank Selection for High-Speed Convolutional Neural Network

Cited by in corpus (8)

• Bayesian Bits: Unifying Quantization and Pruning
• Overfitting for Fun and Profit: Instance-Adaptive Data Compression
• Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
• Learned Threshold Pruning
• A Data and Compute Efficient Design for Limited-Resources Deep Learning
• Structured Convolutions for Efficient Neural Network Design
• Differentiable Joint Pruning and Quantization for Hardware Efficiency
• Cyclic orthogonal convolutions for long-range integration of features