Efficient conformer-based speech recognition with linear attention
arXiv:2104.06865
Abstract
Recently, conformer-based end-to-end automatic speech recognition, which outperforms recurrent neural network based ones, has received much attention. Although the parallel computing of conformer is more efficient than recurrent neural networks, the computational complexity of its dot-product self-attention is quadratic with respect to the length of the input feature. To reduce the computational complexity of the self-attention layer, we propose multi-head linear self-attention for the self-attention layer, which reduces its computational complexity to linear order. In addition, we propose to factorize the feed forward module of the conformer by low-rank matrix factorization, which successfully reduces the number of the parameters by approximate 50% with little performance loss. The proposed model, named linear attention based conformer (LAC), can be trained and inferenced jointly with the connectionist temporal classification objective, which further improves the performance of LAC. To evaluate the effectiveness of LAC, we conduct experiments on the AISHELL-1 and LibriSpeech corpora. Results show that the proposed LAC achieves better performance than 7 recently proposed speech recognition models, and is competitive with the state-of-the-art conformer. Meanwhile, the proposed LAC has a number of parameters of only 50% over the conformer with faster training speed than the latter.
arXiv commentsubmitted to APSIPA ASC 2021
References in corpus (11)
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Sequence Transduction with Recurrent Neural Networks
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Rethinking Attention with Performers
- ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
- Recent Developments on ESPnet Toolkit Boosted by Conformer
- Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attention
- Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
- Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition
- Efficient End-to-End Speech Recognition Using Performers in Conformers