Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling

Abstract

Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible, while full-context ASR waits for the completion of a full speech utterance before emitting completed hypotheses. In this work, we propose a unified framework, Dual-mode ASR, to train a single end-to-end ASR model with shared weights for both streaming and full-context speech recognition. We show that the latency and accuracy of streaming ASR significantly benefit from weight sharing and joint training of full-context ASR, especially with inplace knowledge distillation during the training. The Dual-mode ASR framework can be applied to recent state-of-the-art convolution-based and transformer-based ASR networks. We present extensive experiments with two state-of-the-art ASR networks, ContextNet and Conformer, on two datasets, a widely used public dataset LibriSpeech and a large-scale dataset MultiDomain. Experiments and ablation studies demonstrate that Dual-mode ASR not only simplifies the workflow of training and deploying streaming and full-context ASR models, but also significantly improves both emission latency and recognition accuracy of streaming ASR. With Dual-mode ASR, we achieve new state-of-the-art streaming ASR results on both LibriSpeech and MultiDomain in terms of accuracy and latency.

arXiv commentAccepted in ICLR 2021

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Convolutional Sequence to Sequence Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Attention-Based Models for Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• Weight Uncertainty in Neural Networks
• Searching for Activation Functions
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• Towards Online End-to-end Transformer Automatic Speech Recognition

Cited by in corpus (4)

• TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech Recognition
• ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMs
• Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition
• Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation