Block-Sparse Recurrent Neural Networks

Abstract

Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling. Sparsity is a technique to reduce compute and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms. In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization to create blocks of weights with zeros. Using these techniques, we demonstrate that we can create block-sparse RNNs with sparsity ranging from 80% to 90% with small loss in accuracy. This allows us to reduce the model size by roughly 10x. Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count. Our technique works with a variety of block sizes up to 32x32. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.

References in corpus (9)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Exploring the Limits of Language Modeling
• Learning Structured Sparsity in Deep Neural Networks
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Exploring Sparsity in Recurrent Neural Networks
• Learning Intrinsic Sparse Structures within Long Short-Term Memory

Cited by in corpus (33)

• Recurrent Neural Networks: An Embedded Computing Perspective
• Balanced Sparsity for Efficient DNN Inference on GPU
• Efficient Neural Audio Synthesis
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Accelerating Sparse Deep Neural Networks
• One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
• Rethinking Full Connectivity in Recurrent Neural Networks
• Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity
• FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction
• BRDS: An FPGA-based LSTM Accelerator with Row-Balanced Dual-Ratio Sparsification
• Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM
• Term Revealing: Furthering Quantization at Run Time on Quantized DNNs
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning
• Network Pruning for Low-Rank Binary Indexing
• Fast Training of Sparse Graph Neural Networks on Dense Hardware
• AntMan: Sparse Low-Rank Compression to Accelerate RNN inference
• Towards Structured Dynamic Sparse Pre-Training of BERT
• Tuning Algorithms and Generators for Efficient Edge Inference
• On the convergence of group-sparse autoencoders
• Dynamic Sparsity Neural Networks for Automatic Speech Recognition
• Search Spaces for Neural Model Training
• Cross-Channel Intragroup Sparsity Neural Network
• Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference
• Spectral Pruning for Recurrent Neural Networks
• Full-stack Optimization for Accelerating CNNs with FPGA Validation
• Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Accelerators
• Block-term Tensor Neural Networks
• Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training
• Empirical Evaluation of Deep Learning Model Compression Techniques on the WaveNet Vocoder
• Cascade Weight Shedding in Deep Neural Networks: Benefits and Pitfalls for Network Pruning
• A Survey on Green Deep Learning