Trained Rank Pruning for Efficient Deep Neural Networks

Abstract

To accelerate DNNs inference, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations. Several previous works attempted to directly approximate a pre-trained model by low-rank decomposition; however, small approximation errors in parameters can ripple over a large prediction loss. Apparently, it is not optimal to separate low-rank approximation from training. Unlike previous works, this paper integrates low rank approximation and regularization into the training process. We propose Trained Rank Pruning (TRP), which alternates between low rank approximation and training. TRP maintains the capacity of the original network while imposing low-rank constraints during training. A nuclear regularization optimized by stochastic sub-gradient descent is utilized to further promote low rank in TRP. Networks trained with TRP has a low-rank structure in nature, and is approximated with negligible performance loss, thus eliminating fine-tuning after low rank approximation. The proposed method is comprehensively evaluated on CIFAR-10 and ImageNet, outperforming previous compression counterparts using low rank approximation. Our code is available at: https://github.com/yuhuixu1993/Trained-Rank-Pruning.

overlap with arXiv:1812.02402, in order to merge the two submissions such that withdraw this version

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Wide Residual Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Filters for Efficient ConvNets
• Fixed Point Quantization of Deep Convolutional Networks
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Convolutional neural networks with low-rank regularization
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Efficient and Practical Stochastic Subgradient Descent for Nuclear Norm Regularization
• Accelerating Very Deep Convolutional Networks for Classification and Detection

Cited by in corpus (10)

• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• A flexible, extensible software framework for model compression based on the LC algorithm
• Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification
• BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization
• Principal Component Networks: Parameter Reduction Early in Training
• Refining the Structure of Neural Networks Using Matrix Conditioning
• Improving Network Slimming with Nonconvex Regularization
• Low Rank Regularization: A Review
• A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank
• Compression-aware Continual Learning using Singular Value Decomposition