CNN Acceleration by Low-rank Approximation with Quantized Factors

Abstract

The modern convolutional neural networks although achieve great results in solving complex computer vision tasks still cannot be effectively used in mobile and embedded devices due to the strict requirements for computational complexity, memory and power consumption. The CNNs have to be compressed and accelerated before deployment. In order to solve this problem the novel approach combining two known methods, low-rank tensor approximation in Tucker format and quantization of weights and feature maps (activations), is proposed. The greedy one-step and multi-step algorithms for the task of multilinear rank selection are proposed. The approach for quality restoration after applying Tucker decomposition and quantization is developed. The efficiency of our method is demonstrated for ResNet18 and ResNet34 on CIFAR-10, CIFAR-100 and Imagenet classification tasks. As a result of comparative analysis performed for other methods for compression and acceleration our approach showed its promising features.

References in corpus (30)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Deep Residual Learning for Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• A Survey of Deep Learning Techniques for Autonomous Driving
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Tensor Decompositions for Signal Processing Applications From Two-way to Multiway Component Analysis
• Compressing Deep Convolutional Networks using Vector Quantization
• Predicting Parameters in Deep Learning
• Tensorizing Neural Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Convolutional neural networks with low-rank regularization
• Channel Pruning for Accelerating Very Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Image Super-Resolution Using Very Deep Residual Channel Attention Networks
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Ultimate tensorization: compressing convolutional and FC layers alike
• Fixing the train-test resolution discrepancy: FixEfficientNet
• Towards the Limit of Network Quantization
• Quantized Convolutional Neural Networks for Mobile Devices
• When AWGN-based Denoiser Meets Real Noises
• Matrix and tensor decompositions for training binary neural networks