Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

Abstract

Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resource-limited scenarios. A widely-used practice in relevant work assumes that a smaller-norm parameter or feature plays a less informative role at the inference time. In this paper, we propose a channel pruning technique for accelerating the computations of deep convolutional neural networks (CNNs) that does not critically rely on this assumption. Instead, it focuses on direct simplification of the channel-to-channel computation graph of a CNN without the need of performing a computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structured sparse. Our approach takes two stages: first to adopt an end-to- end stochastic training method that eventually forces the outputs of some channels to be constant, and then to prune those constant channels from the original neural network by adjusting the biases of their impacting layers such that the resulting compact model can be quickly fine-tuned. Our approach is mathematically appealing from an optimization perspective and easy to reproduce. We experimented our approach through several image learning benchmarks and demonstrate its interesting aspects and competitive performance.

arXiv commentaccepted to ICLR 2018, 11 pages

References in corpus (4)

• Learning Structured Sparsity in Deep Neural Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• Learning Efficient Convolutional Networks through Network Slimming
• Data-Driven Sparse Structure Selection for Deep Neural Networks

Cited by in corpus (56)

• Slimmable Neural Networks
• Discrimination-aware Channel Pruning for Deep Neural Networks
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery
• Accelerating Large Scale Real-Time GNN Inference using Channel Pruning
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
• Larger Norm More Transferable: An Adaptive Feature Norm Approach for Unsupervised Domain Adaptation
• Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks
• Data-Driven Sparse Structure Selection for Deep Neural Networks
• STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
• LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Network Pruning That Matters: A Case Study on Retraining Variants
• On the Compression of Neural Networks Using ${\ell }_0$-Norm Regularization and Weight Pruning
• SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• Achieving Adversarial Robustness via Sparsity
• Neural network relief: a pruning algorithm based on neural activity
• Pruning via Iterative Ranking of Sensitivity Statistics
• Patch Slimming for Efficient Vision Transformers
• Rethinking Class-Discrimination Based CNN Channel Pruning
• Filter Grafting for Deep Neural Networks
• SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
• Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Dynamic Sparse Graph for Efficient Deep Learning
• AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• Baseline Pruning-Based Approach to Trojan Detection in Neural Networks
• DASNet: Dynamic Activation Sparsity for Neural Network Efficiency Improvement
• Content-Aware GAN Compression
• Filter Pruning using Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks
• Adaptive Pixel-wise Structured Sparse Network for Efficient CNNs
• Structured Compression by Weight Encryption for Unstructured Pruning and Quantization
• New Perspective of Interpretability of Deep Neural Networks
• Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization
• Channel-wise pruning of neural networks with tapering resource constraint
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
• Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
• How Compact?: Assessing Compactness of Representations through Layer-Wise Pruning
• Channel Planting for Deep Neural Networks using Knowledge Distillation
• KCP: Kernel Cluster Pruning for Dense Labeling Neural Networks
• Out-of-the-box channel pruned networks
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression
• Effective Model Compression via Stage-wise Pruning
• Filter Grafting for Deep Neural Networks: Reason, Method, and Cultivation
• A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank
• Single Image Depth Prediction with Wavelet Decomposition
• Dep-$L_0$: Improving $L_0$-based Network Sparsification via Dependency Modeling
• Class-Discriminative CNN Compression
• Joint Regularization on Activations and Weights for Efficient Neural Network Pruning