Convolutional Neural Network Pruning with Structural Redundancy Reduction

Abstract

Convolutional neural network (CNN) pruning has become one of the most successful network compression approaches in recent years. Existing works on network pruning usually focus on removing the least important filters in the network to achieve compact architectures. In this study, we claim that identifying structural redundancy plays a more essential role than finding unimportant filters, theoretically and empirically. We first statistically model the network pruning problem in a redundancy reduction perspective and find that pruning in the layer(s) with the most structural redundancy outperforms pruning the least important filters across all layers. Based on this finding, we then propose a network pruning approach that identifies structural redundancy of a CNN and prunes filters in the selected layer(s) with the most redundancy. Experiments on various benchmark network architectures and datasets show that our proposed approach significantly outperforms the previous state-of-the-art.

Accepted to CVPR 2021

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Compressing Neural Networks with the Hashing Trick
• Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
• Operation-Aware Soft Channel Pruning using Differentiable Masks
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• CUP: Cluster Pruning for Compressing Deep Neural Networks