SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference

Abstract

In recent years, there has been a flurry of research in deep neural network pruning and compression. Early approaches prune weights individually. However, it is difficult to take advantage of the resulting unstructured sparsity patterns on modern hardware like GPUs. As a result, pruning strategies which impose sparsity structures in the weights have become more popular. However,these structured pruning approaches typically lead to higher losses in accuracy than unstructured pruning. In this paper, we present SparseRT, a code generator that leverage unstructured sparsity to accelerate sparse linear algebra operations in deep learning inference on GPUs. For 1x1 convolutions and fully connected layers, we demonstrate geometric mean of speedups of 3.4x over the equivalent dense computation at 90% sparsity and 5.4x at 95% sparsity when evaluated on hundreds of test cases in deep learning. For sparse 3x3 convolutions, we show speedups of over 5x on use cases in ResNet-50.

Accepted for publication at PACT 2020

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• cuDNN: Efficient Primitives for Deep Learning
• The State of Sparsity in Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• Learning Sparse Neural Networks through $L_0$ Regularization
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• Rigging the Lottery: Making All Tickets Winners
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Balanced Sparsity for Efficient DNN Inference on GPU
• Dissecting the NVidia Turing T4 GPU via Microbenchmarking
• A Closer Look at Structured Pruning for Neural Network Compression
• Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs

Cited by in corpus (4)

• End-to-End Supermask Pruning: Learning to Prune Image Captioning Models
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs
• SSR: Sparse Segment Reduction for Ternary GEMM Acceleration