Online Hybrid CTC/Attention End-to-End Automatic Speech Recognition Architecture

Abstract

Recently, there has been increasing progress in end-to-end automatic speech recognition (ASR) architecture, which transcribes speech to text without any pre-trained alignments. One popular end-to-end approach is the hybrid Connectionist Temporal Classification (CTC) and attention (CTC/attention) based ASR architecture. However, how to deploy hybrid CTC/attention systems for online speech recognition is still a non-trivial problem. This article describes our proposed online hybrid CTC/attention end-to-end ASR architecture, which replaces all the offline components of conventional CTC/attention ASR architecture with their corresponding streaming components. Firstly, we propose stable monotonic chunk-wise attention (sMoChA) to stream the conventional global attention, and further propose monotonic truncated attention (MTA) to simplify sMoChA and solve the training-and-decoding mismatch problem of sMoChA. Secondly, we propose truncated CTC (T-CTC) prefix score to stream CTC prefix score calculation. Thirdly, we design dynamic waiting joint decoding (DWJD) algorithm to dynamically collect the predictions of CTC and attention in an online manner. Finally, we use latency-controlled bidirectional long short-term memory (LC-BLSTM) to stream the widely-used offline bidirectional encoder network. Experiments with LibriSpeech English and HKUST Mandarin tasks demonstrate that, compared with the offline CTC/attention model, our proposed online CTC/attention model improves the real time factor in human-computer interaction services and maintains its performance with moderate degradation. To the best of our knowledge, this is the first work to provide the full-stack online solution for CTC/attention end-to-end ASR architecture.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• ADADELTA: An Adaptive Learning Rate Method
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Attention-Based Models for Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• RNN-T For Latency Controlled ASR With Improved Beam Search
• Towards Online End-to-end Transformer Automatic Speech Recognition
• Semantic Mask for Transformer based End-to-End Speech Recognition

Cited by in corpus (3)

• Transformer-based Online Speech Recognition with Decoder-end Adaptive Computation Steps
• Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuation
• Fast offline Transformer-based end-to-end automatic speech recognition for real-world applications