Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
arXiv:2102.07594
Abstract
Attention-based encoder-decoder (AED) models have achieved promising performance in speech recognition. However, because the decoder predicts text tokens (such as characters or words) in an autoregressive manner, it is difficult for an AED model to predict all tokens in parallel. This makes the inference speed relatively slow. We believe that because the encoder already captures the whole speech utterance, which has the token-level relationship implicitly, we can predict a token without explicitly autoregressive language modeling. When the prediction of a token does not rely on other tokens, the parallel prediction of all tokens in the sequence is realizable. Based on this idea, we propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once). The model consists of an encoder, a decoder, and a position dependent summarizer (PDS). The three modules are based on basic attention blocks. The encoder extracts high-level representations from the speech. The PDS uses positional encodings corresponding to tokens to convert the acoustic representations into token-level representations. The decoder further captures token-level relationships with the self-attention mechanism. At last, the probability distribution on the vocabulary is computed for each token position. Therefore, speech recognition is re-formulated as a position-wise classification problem. Further, we propose a cross-modal transfer learning method to refine semantics from a large-scale pre-trained language model BERT for improving the performance.
arXiv comment14 pages, 7 figures
References in corpus (18)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- Neural Message Passing for Quantum Chemistry
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- FitNets: Hints for Thin Deep Nets
- Attention-Based Models for Speech Recognition
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Sequence Transduction with Recurrent Neural Networks
- A Comparative Study on Transformer vs RNN in Speech Applications
- On Using Monolingual Corpora in Neural Machine Translation
- Online and Linear-Time Attention by Enforcing Monotonic Alignments
- AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale
- Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
- End-to-End ASR-free Keyword Search from Speech
- Self-Attention Transducers for End-to-End Speech Recognition
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
- Graph Structure of Neural Networks
- Unsupervised pre-training for sequence to sequence speech recognition