FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization

Abstract

Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible. However, emitting fast without degrading quality, as measured by word error rate (WER), is highly challenging. Existing approaches including Early and Late Penalties and Constrained Alignments penalize emission delay by manipulating per-token or per-frame probability prediction in sequence transducer models. While being successful in reducing delay, these approaches suffer from significant accuracy regression and also require additional word alignment information from an existing model. In this work, we propose a sequence-level emission regularization method, named FastEmit, that applies latency regularization directly on per-sequence probability in training transducer models, and does not require any alignment. We demonstrate that FastEmit is more suitable to the sequence-level optimization of transducer models for streaming ASR by applying it on various end-to-end streaming ASR networks including RNN-Transducer, Transformer-Transducer, ConvNet-Transducer and Conformer-Transducer. We achieve 150-300 ms latency reduction with significantly better accuracy over previous techniques on a Voice Search test set. FastEmit also improves streaming ASR accuracy from 4.4%/8.9% to 3.1%/7.5% WER, meanwhile reduces 90th percentile latency from 210 ms to only 30 ms on LibriSpeech.

arXiv commentAccepted in ICASSP 2021

References in corpus (10)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Listen, Attend and Spell
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• Low Latency End-to-End Streaming Speech Recognition with a Scout Network
• A comparison of end-to-end models for long-form speech recognition
• Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition

Cited by in corpus (3)

• FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization
• Streaming Multi-talker Speech Recognition with Joint Speaker Identification
• Optimizing Latency for Online Video CaptioningUsing Audio-Visual Transformers