Dual-side Sparse Tensor Core

Abstract

Leveraging sparsity in deep neural network (DNN) models is promising for accelerating model inference. Yet existing GPUs can only leverage the sparsity from weights but not activations, which are dynamic, unpredictable, and hence challenging to exploit. In this work, we propose a novel architecture to efficiently harness the dual-side sparsity (i.e., weight and activation sparsity). We take a systematic approach to understand the (dis)advantages of previous sparsity-related architectures and propose a novel, unexplored paradigm that combines outer-product computation primitive and bitmap-based encoding format. We demonstrate the feasibility of our design with minimal changes to the existing production-scale inner-product-based Tensor Core. We propose a set of novel ISA extensions and co-design the matrix-matrix multiplication and convolution algorithms, which are the two dominant computation patterns in today's DNN models, to exploit our new dual-side sparse Tensor Core. Our evaluation shows that our design can fully unleash the dual-side DNN sparsity and improve the performance by up to one order of magnitude with \hl{small} hardware overhead.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• cuDNN: Efficient Primitives for Deep Learning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Neural Network Distiller: A Python Package For DNN Compression Research
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Accelerating Sparse Deep Neural Networks
• How Far Does BERT Look At:Distance-based Clustering and Analysis of BERT${}^{\prime }$s Attention