Effective Quantization Methods for Recurrent Neural Networks

Abstract

Reducing bit-widths of weights, activations, and gradients of a Neural Network can shrink its storage size and memory usage, and also allow for faster training and inference by exploiting bitwise operations. However, previous attempts for quantization of RNNs show considerable performance degradation when using low bit-width weights and activations. In this paper, we propose methods to quantize the structure of gates and interlinks in LSTM and GRU cells. In addition, we propose balanced quantization methods for weights to further reduce performance degradation. Experiments on PTB and IMDB datasets confirm effectiveness of our methods as performances of our models match or surpass the previous state-of-the-art of quantized RNN.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Recurrent Neural Network Regularization
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Going Deeper with Convolutions
• Compressing Deep Convolutional Networks using Vector Quantization
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Recurrent Neural Networks With Limited Numerical Precision

Cited by in corpus (24)

• Mixed Precision Training
• A Survey on Methods and Theories of Quantized Neural Networks
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Knowledge Distillation from Internal Representations
• Low Precision RNNs: Quantizing RNNs Without Losing Accuracy
• Joint Neural Architecture Search and Quantization
• Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations
• Post-Training 4-bit Quantization on Embedding Tables
• Vau da muntanialas: Energy-efficient multi-die scalable acceleration of RNN inference
• Compression of Acoustic Event Detection Models with Low-rank Matrix Factorization and Quantization Training
• Precision Highway for Ultra Low-Precision Quantization
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Small-Footprint Open-Vocabulary Keyword Spotting with Quantized LSTM Networks
• Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework
• BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization
• Knowledge distillation for optimization of quantized deep neural networks
• Histogram-Equalized Quantization for logic-gated Residual Neural Networks
• Edge Intelligence Empowered UAVs for Automated Wind Farm Monitoring in Smart Grids
• Compression of Acoustic Event Detection Models With Quantized Distillation
• 4-bit Quantization of LSTM-based Speech Recognition Models
• AutoQNN: An End-to-End Framework for Automatically Quantizing Neural Networks
• A Survey on Green Deep Learning