Automated Pruning for Deep Neural Network Compression

Abstract

In this work we present a method to improve the pruning step of the current state-of-the-art methodology to compress neural networks. The novelty of the proposed pruning technique is in its differentiability, which allows pruning to be performed during the backpropagation phase of the network training. This enables an end-to-end learning and strongly reduces the training time. The technique is based on a family of differentiable pruning functions and a new regularizer specifically designed to enforce pruning. The experimental results show that the joint optimization of both the thresholds and the network weights permits to reach a higher compression rate, reducing the number of weights of the pruned network by a further 14% to 33% compared to the current state-of-the-art. Furthermore, we believe that this is the first study where the generalization capabilities in transfer learning tasks of the features extracted by a pruned network are analyzed. To achieve this goal, we show that the representations learned using the proposed pruning methodology maintain the same effectiveness and generality of those learned by the corresponding non-compressed network on a set of different recognition tasks.

8 pages, 5 figures. Published as a conference paper at ICPR 2018

References in corpus (6)

• How transferable are features in deep neural networks?
• Theoretical Models of Learning to Learn
• Compressing Deep Convolutional Networks using Vector Quantization
• Opening the Black Box of Deep Neural Networks via Information
• Compressing Neural Networks with the Hashing Trick
• Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields

Cited by in corpus (10)

• Dynamic Graph Convolutional Networks
• Edge Intelligence: Architectures, Challenges, and Applications
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Learned Threshold Pruning
• Constructing Deep Neural Networks by Bayesian Network Structure Learning
• Structured Convolutions for Efficient Neural Network Design
• Automated Model Compression by Jointly Applied Pruning and Quantization
• Knowledge Distillation By Sparse Representation Matching
• Solving hybrid machine learning tasks by traversing weight space geodesics
• Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning