Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attention

Abstract

Recently, several studies reported that dot-product selfattention (SA) may not be indispensable to the state-of-theart Transformer models. Motivated by the fact that dense synthesizer attention (DSA), which dispenses with dot products and pairwise interactions, achieved competitive results in many language processing tasks, in this paper, we first propose a DSA-based speech recognition, as an alternative to SA. To reduce the computational complexity and improve the performance, we further propose local DSA (LDSA) to restrict the attention scope of DSA to a local range around the current central frame for speech recognition. Finally, we combine LDSA with SA to extract the local and global information simultaneously. Experimental results on the Ai-shell1 Mandarine speech recognition corpus show that the proposed LDSA-Transformer achieves a character error rate (CER) of 6.49%, which is slightly better than that of the SA-Transformer. Meanwhile, the LDSA-Transformer requires less computation than the SATransformer. The proposed combination method not only achieves a CER of 6.18%, which significantly outperforms the SA-Transformer, but also has roughly the same number of parameters and computational complexity as the latter. The implementation of the multi-head LDSA is available at https://github.com/mlxu995/multihead-LDSA.

arXiv comment5 pages, 3 figures

References in corpus (11)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• A Comparative Study on Transformer vs RNN in Speech Applications
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Lite Transformer with Long-Short Range Attention
• Self-Attention Transducers for End-to-End Speech Recognition
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition

Cited by in corpus (7)

• A Multi-grained based Attention Network for Semi-supervised Sound Event Detection
• Efficient conformer-based speech recognition with linear attention
• Efficient End-to-End Speech Recognition Using Performers in Conformers
• Transformer-based end-to-end speech recognition with residual Gaussian-based self-attention
• U-shaped Transformer with Frequency-Band Aware Attention for Speech Enhancement
• Efficient Transformer for Direct Speech Translation
• Conformer-based End-to-end Speech Recognition With Rotary Position Embedding