Dynamic Network Surgery for Efficient DNNs

Abstract

Deep learning has become a ubiquitous technology to improve machine intelligence. However, most of the existing deep models are structurally very complex, making them difficult to be deployed on the mobile platforms with limited computational power. In this paper, we propose a novel network compression method called dynamic network surgery, which can remarkably reduce the network complexity by making on-the-fly connection pruning. Unlike the previous methods which accomplish this task in a greedy way, we properly incorporate connection splicing into the whole process to avoid incorrect pruning and make it as a continual network maintenance. The effectiveness of our method is proved with experiments. Without any accuracy loss, our method can efficiently compress the number of parameters in LeNet-5 and AlexNet by a factor of $108\times$ and $17.7\times$ respectively, proving that it outperforms the recent pruning method by considerable margins. Code and some models are available at https://github.com/yiwenguo/Dynamic-Network-Surgery.

Accepted by NIPS 2016

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• Fast Training of Convolutional Networks through FFTs

Cited by in corpus (19)

• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• MaskConnect: Connectivity Learning by Gradient Descent
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Rethinking Weight Decay For Efficient Neural Network Pruning
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• FreezeNet: Full Performance by Reduced Storage Costs
• Taxonomy of Saliency Metrics for Channel Pruning
• Backdoor Attacks on Federated Learning with Lottery Ticket Hypothesis
• High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• Block-wise Dynamic Sparseness
• Channel-wise pruning of neural networks with tapering resource constraint
• AutoPruning for Deep Neural Network with Dynamic Channel Masking
• ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition