Coordinating Filters for Faster Deep Neural Networks
arXiv:1703.09746
Abstract
Very large-scale Deep Neural Networks (DNNs) have achieved remarkable successes in a large variety of computer vision tasks. However, the high computation intensity of DNNs makes it challenging to deploy these models on resource-limited systems. Some studies used low-rank approaches that approximate the filters by low-rank basis to accelerate the testing. Those works directly decomposed the pre-trained DNNs by Low-Rank Approximations (LRA). How to train DNNs toward lower-rank space for more efficient DNNs, however, remains as an open area. To solve the issue, in this work, we propose Force Regularization, which uses attractive forces to enforce filters so as to coordinate more weight information into lower-rank space. We mathematically and empirically verify that after applying our technique, standard LRA methods can reconstruct filters using much lower basis and thus result in faster DNNs. The effectiveness of our approach is comprehensively evaluated in ResNets, AlexNet, and GoogLeNet. In AlexNet, for example, Force Regularization gains 2x speedup on modern GPU without accuracy loss and 4.05x speedup on CPU by paying small accuracy degradation. Moreover, Force Regularization better initializes the low-rank DNNs such that the fine-tuning can converge faster toward higher accuracy. The obtained lower-rank DNNs can be further sparsified, proving that Force Regularization can be integrated with state-of-the-art sparsity-based acceleration methods. Source code is available in https://github.com/wenwei202/caffe
arXiv commentICCV 2017
References in corpus (15)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Residual Learning for Image Recognition
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Going Deeper with Convolutions
- Predicting Parameters in Deep Learning
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Learning Structured Sparsity in Deep Neural Networks
- Convolutional neural networks with low-rank regularization
- Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning
- Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
- Fast Algorithms for Convolutional Neural Networks
- k-Means Clustering Is Matrix Factorization