SparseDNN: Fast Sparse Deep Learning Inference on CPUs
arXiv:2101.07948
Abstract
The last few years have seen gigantic leaps in algorithms and systems to support efficient deep learning inference. Pruning and quantization algorithms can now consistently compress neural networks by an order of magnitude. For a compressed neural network, a multitude of inference frameworks have been designed to maximize the performance of the target hardware. While we find mature support for quantized neural networks in production frameworks such as OpenVINO and MNN, support for pruned sparse neural networks is still lacking. To tackle this challenge, we present SparseDNN, a sparse deep learning inference engine targeting CPUs. We present both kernel-level optimizations with a sparse code generator to accelerate sparse operators and novel network-level optimizations catering to sparse networks. We show that our sparse code generator can achieve significant speedups over state-of-the-art sparse and dense libraries. On end-to-end benchmarks such as Huggingface pruneBERT, SparseDNN achieves up to 5x throughput improvement over dense inference with state-of-the-art OpenVINO. Open source library at: https://github.com/marsupialtail/sparsednn.
References in corpus (21)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Gaussian Error Linear Units (GELUs)
- Language Models are Few-Shot Learners
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- cuDNN: Efficient Primitives for Deep Learning
- The State of Sparsity in Deep Neural Networks
- Variational Dropout Sparsifies Deep Neural Networks
- Learning Sparse Neural Networks through Regularization
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning
- Rigging the Lottery: Making All Tickets Winners
- CondenseNet: An Efficient DenseNet using Learned Group Convolutions
- Balanced Sparsity for Efficient DNN Inference on GPU
- Optimizing CNN Model Inference on CPUs
- Movement Pruning: Adaptive Sparsity by Fine-Tuning
- A Closer Look at Structured Pruning for Neural Network Compression
- Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs
- SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
- High-Performance Deep Learning via a Single Building Block
- MNN: A Universal and Efficient Inference Engine