Cross-Channel Intragroup Sparsity Neural Network

Abstract

Modern deep neural networks rely on overparameterization to achieve state-of-the-art generalization. But overparameterized models are computationally expensive. Network pruning is often employed to obtain less demanding models for deployment. Fine-grained pruning removes individual weights in parameter tensors and can achieve a high model compression ratio with little accuracy degradation. However, it introduces irregularity into the computing dataflow and often does not yield improved model inference efficiency in practice. Coarse-grained model pruning, while realizing satisfactory inference speedup through removal of network weights in groups, e.g. an entire filter, often lead to significant accuracy degradation. This work introduces the cross-channel intragroup (CCI) sparsity structure, which can prevent the inference inefficiency of fine-grained pruning while maintaining outstanding model performance. We then present a novel training algorithm designed to perform well under the constraint imposed by the CCI-Sparsity. Through a series of comparative experiments we show that our proposed CCI-Sparsity structure and the corresponding pruning algorithm outperform prior art in inference efficiency by a substantial margin given suited hardware acceleration in the future.

References in corpus (14)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Residual Learning for Image Recognition
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• cuDNN: Efficient Primitives for Deep Learning
• Pruning Filters for Efficient ConvNets
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Learning Structured Sparsity in Deep Neural Networks
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Block-Sparse Recurrent Neural Networks
• More is Less: A More Complicated Network with Less Inference Complexity
• ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
• IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks