A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
arXiv:1911.03604
Abstract
While significant improvements have been made in recent years in terms of end-to-end automatic speech recognition (ASR) performance, such improvements were obtained through the use of very large neural networks, unfit for embedded use on edge devices. That being said, in this paper, we work on simplifying and compressing Transformer-based encoder-decoder architectures for the end-to-end ASR task. We empirically introduce a more compact Speech-Transformer by investigating the impact of discarding particular modules on the performance of the model. Moreover, we evaluate reducing the numerical precision of our network's weights and activations while maintaining the performance of the full-precision model. Our experiments show that we can reduce the number of parameters of the full-precision model and then further compress the model 4x by fully quantizing to 8-bit fixed point precision.
arXiv commentSubmitted to IEEE Signal Processing Letters Minor changes in Section 3
References in corpus (12)
- Sequence to Sequence Learning with Neural Networks
- ADADELTA: An Adaptive Learning Rate Method
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Sequence Transduction with Recurrent Neural Networks
- A Comparative Study on Transformer vs RNN in Speech Applications
- Q8BERT: Quantized 8Bit BERT
- Model compression via distillation and quantization
- Language Modeling with Deep Transformers
- Transformers with convolutional context for ASR
- Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model
- Streaming End-to-end Speech Recognition For Mobile Devices