Low Latency End-to-End Streaming Speech Recognition with a Scout Network

Abstract

The attention-based Transformer model has achieved promising results for speech recognition (SR) in the offline mode. However, in the streaming mode, the Transformer model usually incurs significant latency to maintain its recognition accuracy when applying a fixed-length look-ahead window in each encoder layer. In this paper, we propose a novel low-latency streaming approach for Transformer models, which consists of a scout network and a recognition network. The scout network detects the whole word boundary without seeing any future frames, while the recognition network predicts the next subword by utilizing the information from all the frames before the predicted boundary. Our model achieves the best performance (2.7/6.4 WER) with only 639 ms latency on the test-clean and test-other data sets of Librispeech.

References in corpus (11)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Attention-Based Models for Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• A Comparative Study on Transformer vs RNN in Speech Applications
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
• Towards Online End-to-end Transformer Automatic Speech Recognition
• Semantic Mask for Transformer based End-to-End Speech Recognition
• An Online Attention-based Model for Speech Recognition
• Synchronous Transformers for End-to-End Speech Recognition

Cited by in corpus (11)

• Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition
• Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition
• Parallelizing Legendre Memory Unit Training
• FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
• Multi-view Frequency LSTM: An Efficient Frontend for Automatic Speech Recognition
• A Better and Faster End-to-End Model for Streaming ASR
• Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data
• Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios
• Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR
• Dissecting User-Perceived Latency of On-Device E2E Speech Recognition
• AV Taris: Online Audio-Visual Speech Recognition