Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition

Abstract

This paper proposes an efficient memory transformer Emformer for low latency streaming speech recognition. In Emformer, the long-range history context is distilled into an augmented memory bank to reduce self-attention's computation complexity. A cache mechanism saves the computation for the key and value in self-attention for the left context. Emformer applies a parallelized block processing in training to support low latency models. We carry out experiments on benchmark LibriSpeech data. Under average latency of 960 ms, Emformer gets WER $2.50\%$ on test-clean and $5.62\%$ on test-other. Comparing with a strong baseline augmented memory transformer (AM-TRF), Emformer gets $4.6$ folds training speedup and $18\%$ relative real-time factor (RTF) reduction in decoding with relative WER reduction $17\%$ on test-clean and $9\%$ on test-other. For a low latency scenario with an average latency of 80 ms, Emformer achieves WER $3.01\%$ on test-clean and $7.09\%$ on test-other. Comparing with the LSTM baseline with the same latency and model size, Emformer gets relative WER reduction $9\%$ and $16\%$ on test-clean and test-other, respectively.

arXiv comment5 pages, 2 figures, submitted to ICASSP 2021

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese
• Low Latency End-to-End Streaming Speech Recognition with a Scout Network
• From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition

Cited by in corpus (5)

• Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset
• SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts
• Efficient End-to-End Speech Recognition Using Performers in Conformers
• Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications
• Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning