Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights

Abstract

Machine learning (ML) models are widely used in many important domains. For efficiently processing these computational- and memory-intensive applications, tensors of these over-parameterized models are compressed by leveraging sparsity, size reduction, and quantization of tensors. Unstructured sparsity and tensors with varying dimensions yield irregular computation, communication, and memory access patterns; processing them on hardware accelerators in a conventional manner does not inherently leverage acceleration opportunities. This paper provides a comprehensive survey on the efficient execution of sparse and irregular tensor computations of ML models on hardware accelerators. In particular, it discusses enhancement modules in the architecture design and the software support; categorizes different hardware designs and acceleration techniques and analyzes them in terms of hardware and execution costs; analyzes achievable accelerations for recent DNNs; highlights further opportunities in terms of hardware/software/model co-design optimizations (inter/intra-module). The takeaways from this paper include: understanding the key challenges in accelerating sparse, irregular-shaped, and quantized tensors; understanding enhancements in accelerator systems for supporting their efficient computations; analyzing trade-offs in opting for a specific design choice for encoding, storing, extracting, communicating, computing, and load-balancing the non-zeros; understanding how structured sparsity can improve storage efficiency and balance computations; understanding how to compile and map models with sparse tensors on the accelerators; understanding recent design trends for efficient accelerations and further opportunities.

Accepted for publication in Proceedings of the IEEE

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Sequence to Sequence Learning with Neural Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Language Models are Few-Shot Learners
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Learning with Limited Numerical Precision
• One weird trick for parallelizing convolutional neural networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• The State of Sparsity in Deep Neural Networks
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Device Placement Optimization with Reinforcement Learning
• Benchmarking TinyML Systems: Challenges and Direction
• CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
• GraphACT: Accelerating GCN Training on CPU-FPGA Heterogeneous Platforms
• MLIR: A Compiler Infrastructure for the End of Moore's Law
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• A Study of BFLOAT16 for Deep Learning Training
• ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks
• Neural Network Distiller: A Python Package For DNN Compression Research
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• CSB-RNN: A Faster-than-Realtime RNN Acceleration Framework with Compressed Structured Blocks
• Programmatic Control of a Compiler for Generating High-performance Spatial Hardware

Cited by in corpus (9)

• Machine Learning for Microcontroller-Class Hardware: A Review
• Capstan: A Vector RDA for Sparsity
• Sparse Stream Semantic Registers: A Lightweight ISA Extension Accelerating General Sparse Linear Algebra
• A Configurable BNN ASIC using a Network of Programmable Threshold Logic Standard Cells
• Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse Multi-DNN Workloads
• D-SELD: Dataset-Scalable Exemplar LCA-Decoder
• The Magnificent Seven Challenges and Opportunities in Domain-Specific Accelerator Design for Autonomous Systems
• Sparse Periodic Systolic Dataflow for Lowering Latency and Power Dissipation of Convolutional Neural Network Accelerators
• Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems