Penetrating the Fog: the Path to Efficient CNN Models

Abstract

With the increasing demand to deploy convolutional neural networks (CNNs) on mobile platforms, the sparse kernel approach was proposed, which could save more parameters than the standard convolution while maintaining accuracy. However, despite the great potential, no prior research has pointed out how to craft an sparse kernel design with such potential (i.e., effective design), and all prior works just adopt simple combinations of existing sparse kernels such as group convolution. Meanwhile due to the large design space it is also impossible to try all combinations of existing sparse kernels. In this paper, we are the first in the field to consider how to craft an effective sparse kernel design by eliminating the large design space. Specifically, we present a sparse kernel scheme to illustrate how to reduce the space from three aspects. First, in terms of composition we remove designs composed of repeated layers. Second, to remove designs with large accuracy degradation, we find an unified property named information field behind various sparse kernel designs, which could directly indicate the final accuracy. Last, we remove designs in two cases where a better parameter efficiency could be achieved. Additionally, we provide detailed efficiency analysis on the final four designs in our scheme. Experimental results validate the idea of our scheme by showing that our scheme is able to find designs which are more efficient in using parameters and computation with similar or higher accuracy.

arXiv comment13 pages, 1 figure

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Residual Learning for Image Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Densely Connected Convolutional Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Mixed Precision Training
• Pruning Filters for Efficient ConvNets
• Tensorizing Neural Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• Ultimate tensorization: compressing convolutional and FC layers alike
• Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks