Recent Advances in Efficient Computation of Deep Convolutional Neural Networks

Abstract

Deep neural networks have evolved remarkably over the past few years and they are currently the fundamental tools of many intelligent systems. At the same time, the computational complexity and resource consumption of these networks also continue to increase. This will pose a significant challenge to the deployment of such networks, especially in real-time applications or on resource-limited devices. Thus, network acceleration has become a hot topic within the deep learning community. As for hardware implementation of deep neural networks, a batch of accelerators based on FPGA/ASIC have been proposed in recent years. In this paper, we provide a comprehensive survey of recent advances in network acceleration, compression and accelerator design from both algorithm and hardware points of view. Specifically, we provide a thorough analysis of each of the following topics: network pruning, low-rank approximation, network quantization, teacher-student networks, compact network design and hardware accelerators. Finally, we will introduce and discuss a few possible future directions.

arXiv comment14 pages, 3 figures

References in corpus (25)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• FitNets: Hints for Thin Deep Nets
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Ternary Weight Networks
• Trained Ternary Quantization
• Predicting Parameters in Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks
• Convolutional Neural Networks using Logarithmic Data Representation
• Channel Pruning for Accelerating Very Deep Neural Networks
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Bitwise Neural Networks
• Neural Networks with Few Multiplications
• 8-Bit Approximations for Parallelism in Deep Learning
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks
• Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation

Cited by in corpus (2)

• An Experimental Study of the Impact of Pre-training on the Pruning of a Convolutional Neural Network
• Post-training Quantization with Multiple Points: Mixed Precision without Mixed Precision