4-bit Quantization of LSTM-based Speech Recognition Models

Abstract

We investigate the impact of aggressive low-precision representations of weights and activations in two families of large LSTM-based architectures for Automatic Speech Recognition (ASR): hybrid Deep Bidirectional LSTM - Hidden Markov Models (DBLSTM-HMMs) and Recurrent Neural Network - Transducers (RNN-Ts). Using a 4-bit integer representation, a naïve quantization approach applied to the LSTM portion of these models results in significant Word Error Rate (WER) degradation. On the other hand, we show that minimal accuracy loss is achievable with an appropriate choice of quantizers and initializations. In particular, we customize quantization schemes depending on the local properties of the network, improving recognition performance while limiting computational time. We demonstrate our solution on the Switchboard (SWB) and CallHome (CH) test sets of the NIST Hub5-2000 evaluation. DBLSTM-HMMs trained with 300 or 2000 hours of SWB data achieves $<$0.5% and $<$1% average WER degradation, respectively. On the more challenging RNN-T models, our quantization strategy limits degradation in 4-bit inference to 1.3%.

arXiv comment5 pages, 3 figures, Andrea Fasoli and Chia-Yu Chen equally contributed to this work. Paper accepted to Interspeech 2021

References in corpus (9)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Effective Quantization Methods for Recurrent Neural Networks
• Optimizing Speech Recognition For The Edge
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)