A Closer Look at Structured Pruning for Neural Network Compression

Abstract

Structured pruning is a popular method for compressing a neural network: given a large trained network, one alternates between removing channel connections and fine-tuning; reducing the overall width of the network. However, the efficacy of structured pruning has largely evaded scrutiny. In this paper, we examine ResNets and DenseNets obtained through structured pruning-and-tuning and make two interesting observations: (i) reduced networks---smaller versions of the original network trained from scratch---consistently outperform pruned networks; (ii) if one takes the architecture of a pruned network and then trains it from scratch it is significantly more competitive. Furthermore, these architectures are easy to approximate: we can prune once and obtain a family of new, scalable network architectures that can simply be trained from scratch. Finally, we compare the inference speed of reduced and pruned networks on hardware, and show that reduced networks are significantly faster. Code is available at https://github.com/BayesWatch/pytorch-prunes.

arXiv commentPreprint. First two authors contributed equally. Paper title has changed

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Do Deep Nets Really Need to be Deep?
• DARTS: Differentiable Architecture Search
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Rethinking the Value of Network Pruning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Group Sparse Regularization for Deep Neural Networks
• The State of Sparsity in Deep Neural Networks
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
• Faster gaze prediction with dense networks and Fisher pruning
• Critical Learning Periods in Deep Neural Networks
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• Separable Layers Enable Structured Efficient Linear Substitutions

Cited by in corpus (6)

• Joint Multi-Dimension Pruning via Numerical Gradient Update
• SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
• Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs
• Dirichlet Pruning for Neural Network Compression
• Reproducibility Study: Comparing Rewinding and Fine-tuning in Neural Network Pruning