Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN

Abstract

The existence of redundancy in Convolutional Neural Networks (CNNs) enables us to remove some filters/channels with acceptable performance drops. However, the training objective of CNNs usually tends to minimize an accuracy-related loss function without any attention paid to the redundancy, making the redundancy distribute randomly on all the filters, such that removing any of them may trigger information loss and accuracy drop, necessitating a following finetuning step for recovery. In this paper, we propose to manipulate the redundancy during training to facilitate network pruning. To this end, we propose a novel Centripetal SGD (C-SGD) to make some filters identical, resulting in ideal redundancy patterns, as such filters become purely redundant due to their duplicates; hence removing them does not harm the network. As shown on CIFAR and ImageNet, C-SGD delivers better performance because the redundancy is better organized, compared to the existing methods. The efficiency also characterizes C-SGD because it is as fast as regular SGD, requires no finetuning, and can be conducted simultaneously on all the layers even in very deep CNNs. Besides, C-SGD can improve the accuracy of CNNs by first training a model with the same architecture but wider layers then squeezing it into the original width.

arXiv commentExtension of the CVPR-2019 paper (https://openaccess.thecvf.com/content_CVPR_2019/papers/Ding_Centripetal_SGD_for_Pruning_Very_Deep_Convolutional_Networks_With_Complicated_CVPR_2019_paper.pdf). arXiv admin note: substantial text overlap with arXiv:1904.03837

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Learning the Number of Neurons in Deep Networks
• Memory Bounded Deep Convolutional Networks
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
• Structured Transforms for Small-Footprint Deep Learning
• A novel channel pruning method for deep neural network compression
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression
• Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge Devices
• Structural Compression of Convolutional Neural Networks
• Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning
• Accelerate CNN via Recursive Bayesian Pruning