Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights

Abstract

Machine learning (ML) models are widely used in many important domains. For efficiently processing these computational- and memory-intensive applications, tensors of these over-parameterized models are compressed by leveraging sparsity, size reduction, and quantization of tensors. Unstructured sparsity and tensors with varying dimensions yield irregular computation, communication, and memory access patterns; processing them on hardware accelerators in a conventional manner does not inherently leverage acceleration opportunities. This paper provides a comprehensive survey on the efficient execution of sparse and irregular tensor computations of ML models on hardware accelerators. In particular, it discusses enhancement modules in the architecture design and the software support; categorizes different hardware designs and acceleration techniques and analyzes them in terms of hardware and execution costs; analyzes achievable accelerations for recent DNNs; highlights further opportunities in terms of hardware/software/model co-design optimizations (inter/intra-module). The takeaways from this paper include: understanding the key challenges in accelerating sparse, irregular-shaped, and quantized tensors; understanding enhancements in accelerator systems for supporting their efficient computations; analyzing trade-offs in opting for a specific design choice for encoding, storing, extracting, communicating, computing, and load-balancing the non-zeros; understanding how structured sparsity can improve storage efficiency and balance computations; understanding how to compile and map models with sparse tensors on the accelerators; understanding recent design trends for efficient accelerations and further opportunities.

arXiv commentAccepted for publication in Proceedings of the IEEE

References in corpus (69)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Survey on Deep Learning in Medical Image Analysis
• Sequence to Sequence Learning with Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: A system for large-scale machine learning
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• YOLOv3: An Incremental Improvement
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• End to End Learning for Self-Driving Cars
• Language Models are Few-Shot Learners
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Neural Architecture Search: A Survey
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Graph Neural Networks: A Review of Methods and Applications
• Tensor Decompositions for Signal Processing Applications From Two-way to Multiway Component Analysis
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Deep Learning with Limited Numerical Precision
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• One weird trick for parallelizing convolutional neural networks
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Fixed Point Quantization of Deep Convolutional Networks
• FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• The State of Sparsity in Deep Neural Networks
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Convolutional Neural Networks using Logarithmic Data Representation
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• Device Placement Optimization with Reinforcement Learning
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• Benchmarking TinyML Systems: Challenges and Direction
• CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
• Survey and Benchmarking of Machine Learning Accelerators
• Exploring Sparsity in Recurrent Neural Networks
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• Recurrent Neural Networks: An Embedded Computing Perspective
• Format Abstraction for Sparse Tensor Algebra Compilers
• GraphACT: Accelerating GCN Training on CPU-FPGA Heterogeneous Platforms
• MLIR: A Compiler Infrastructure for the End of Moore's Law
• Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
• Accelerator-Aware Pruning for Convolutional Neural Networks
• Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference
• Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• AutoDNNchip: An Automated DNN Chip Predictor and Builder for Both FPGAs and ASICs
• A Study of BFLOAT16 for Deep Learning Training
• ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks
• Neural Network Distiller: A Python Package For DNN Compression Research
• Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks
• Cnvlutin2: Ineffectual-Activation-and-Weight-Free Deep Neural Network Computing
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• DLVM: A modern compiler infrastructure for deep learning systems
• Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?
• CSB-RNN: A Faster-than-Realtime RNN Acceleration Framework with Compressed Structured Blocks
• Programmatic Control of a Compiler for Generating High-performance Spatial Hardware
• Hierarchical Block Sparse Neural Networks
• Faster Neural Network Training with Approximate Tensor Operations
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• DASNet: Dynamic Activation Sparsity for Neural Network Efficiency Improvement

Cited by in corpus (9)

• Machine Learning for Microcontroller-Class Hardware: A Review
• Capstan: A Vector RDA for Sparsity
• Sparse Stream Semantic Registers: A Lightweight ISA Extension Accelerating General Sparse Linear Algebra
• A Configurable BNN ASIC using a Network of Programmable Threshold Logic Standard Cells
• Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse Multi-DNN Workloads
• D-SELD: Dataset-Scalable Exemplar LCA-Decoder
• The Magnificent Seven Challenges and Opportunities in Domain-Specific Accelerator Design for Autonomous Systems
• Sparse Periodic Systolic Dataflow for Lowering Latency and Power Dissipation of Convolutional Neural Network Accelerators
• Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems