Convolutional neural networks compression with low rank and sparse tensor decompositions

Abstract

Convolutional neural networks show outstanding results in a variety of computer vision tasks. However, a neural network architecture design usually faces a trade-off between model performance and computational/memory complexity. For some real-world applications, it is crucial to develop models, which can be fast and light enough to run on edge systems and mobile devices. However, many modern architectures that demonstrate good performance don't satisfy inference time and storage limitation requirements. Thus, arises a problem of neural network compression to obtain a smaller and faster model, which is on par with the initial one. In this work, we consider a neural network compression method based on tensor decompositions. Namely, we propose to approximate the convolutional layer weight with a tensor, which can be represented as a sum of low-rank and sparse components. The motivation for such approximation is based on the assumption that low-rank and sparse terms allow eliminating two different types of redundancy and thus yield a better compression rate. An efficient CPU implementation for the proposed method has been developed. Our algorithm has demonstrated up to 3.5x CPU layer speedup and 11x layer size reduction when compressing Resnet50 architecture for the image classification task.

References in corpus (24)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Rethinking Atrous Convolution for Semantic Image Segmentation
• YOLOv3: An Incremental Improvement
• Densely Connected Convolutional Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Going Deeper with Convolutions
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Learning Structured Sparsity in Deep Neural Networks
• The State of Sparsity in Deep Neural Networks
• Convolutional neural networks with low-rank regularization
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Learning Efficient Convolutional Networks through Network Slimming
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Faster gaze prediction with dense networks and Fisher pruning
• XNOR-Net++: Improved Binary Neural Networks
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• Quantized Convolutional Neural Networks for Mobile Devices
• Robust PCA for Anomaly Detection in Cyber Networks
• LCA: Loss Change Allocation for Neural Network Training
• Differentiable Sparsification for Deep Neural Networks
• Matrix and tensor decompositions for training binary neural networks
• Error Preserving Correction for CPD and Bounded-Norm CPD