Balanced Sparsity for Efficient DNN Inference on GPU

Abstract

In trained deep neural networks, unstructured pruning can reduce redundant weights to lower storage cost. However, it requires the customization of hardwares to speed up practical inference. Another trend accelerates sparse model inference on general-purpose hardwares by adopting coarse-grained sparsity to prune or regularize consecutive weights for efficient computation. But this method often sacrifices model accuracy. In this paper, we propose a novel fine-grained sparsity approach, balanced sparsity, to achieve high model accuracy with commercial hardwares efficiently. Our approach adapts to high parallelism property of GPU, showing incredible potential for sparsity in the widely deployment of deep learning services. Experiment results show that balanced sparsity achieves up to 3.1x practical speedup for model inference on GPU, while retains the same high model accuracy as fine-grained sparsity.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Block-Sparse Recurrent Neural Networks
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee

Cited by in corpus (16)

• Big Bird: Transformers for Longer Sequences
• What is the State of Neural Network Pruning?
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Tight Compression: Compressing CNN Through Fine-Grained Pruning and Weight Permutation for Efficient Implementation
• ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning
• The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
• Accelerating Sparse Deep Neural Networks
• Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity
• FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction
• SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs
• Dual-side Sparse Tensor Core
• Accelerating convolutional neural network by exploiting sparsity on GPUs
• Empirical Evaluation of Deep Learning Model Compression Techniques on the WaveNet Vocoder
• Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning
• Graph Conditioned Sparse-Attention for Improved Source Code Understanding