Dynamic Channel Pruning: Feature Boosting and Suppression

Abstract

Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources. In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time. FBS introduces small auxiliary connections to existing convolutional layers. In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels. FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs. We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification. Experiments show that FBS can respectively provide $5\times$ and $2\times$ savings in compute on VGG-16 and ResNet-18, both with less than $0.6\%$ top-5 accuracy loss.

arXiv comment14 pages, 5 figures, 4 tables, published as a conference paper at ICLR 2019

References in corpus (6)

• Channel Pruning for Accelerating Very Deep Neural Networks
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Channel Gating Neural Networks
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
• More is Less: A More Complicated Network with Less Inference Complexity
• Changing Model Behavior at Test-Time Using Reinforcement Learning

Cited by in corpus (36)

• Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
• Bringing AI To Edge: From Deep Learning's Perspective
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• Human Activity Recognition on Microcontrollers with Quantized and Adaptive Deep Neural Networks
• Batch-Shaping for Learning Conditional Channel Gated Networks
• Tensor Decomposition for Model Reduction in Neural Networks: A Review
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Locally Free Weight Sharing for Network Width Search
• AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications
• AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
• STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• EREBA: Black-box Energy Testing of Adaptive Neural Networks
• Rethinking Class-Discrimination Based CNN Channel Pruning
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• Faster Neural Network Training with Approximate Tensor Operations
• Manipulating SGD with Data Ordering Attacks
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware
• FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Training
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• Dynamic Group Convolution for Accelerating Convolutional Neural Networks
• VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Attentive Feature Reuse for Multi Task Meta learning
• Split to Be Slim: An Overlooked Redundancy in Vanilla Convolution
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Block-wise Dynamic Sparseness
• Example-Based Concept Analysis Framework for Deep Weather Forecast Models
• Adaptive Pixel-wise Structured Sparse Network for Efficient CNNs
• Learning Instance-wise Sparsity for Accelerating Deep Models
• Dynamic Slimmable Network
• AntiDote: Attention-based Dynamic Optimization for Neural Network Runtime Efficiency
• Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better
• Third ArchEdge Workshop: Exploring the Design Space of Efficient Deep Neural Networks
• Channel Pruning Guided by Spatial and Channel Attention for DNNs in Intelligent Edge Computing