Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognition

Abstract

Recently very deep transformers have outperformed conventional bi-directional long short-term memory networks by a large margin in speech recognition. However, to put it into production usage, inference computation cost is still a serious concern in real scenarios. In this paper, we study two different non-autoregressive transformer structure for automatic speech recognition (ASR): A-CMLM and A-FMLM. During training, for both frameworks, input tokens fed to the decoder are randomly replaced by special mask tokens. The network is required to predict the tokens corresponding to those mask tokens by taking both unmasked context and input speech into consideration. During inference, we start from all mask tokens and the network iteratively predicts missing tokens based on partial results. We show that this framework can support different decoding strategies, including traditional left-to-right. A new decoding strategy is proposed as an example, which starts from the easiest predictions to the most difficult ones. Results on Mandarin (Aishell) and Japanese (CSJ) ASR benchmarks show the possibility to train such a non-autoregressive network for ASR. Especially in Aishell, the proposed method outperformed the Kaldi ASR system and it matches the performance of the state-of-the-art autoregressive transformer with 7x speedup. Pretrained models and code will be made available after publication.

References in corpus (12)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Attention-Based Models for Speech Recognition
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• A Comparative Study on Transformer vs RNN in Speech Applications
• Non-Autoregressive Neural Machine Translation
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Non-Monotonic Sequential Text Generation
• Unsupervised pre-training for sequence to sequence speech recognition
• CAT: CRF-based ASR Toolkit
• Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition

Cited by in corpus (11)

• A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition
• Non-autoregressive Transformer-based End-to-end ASR using BERT
• TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech Recognition
• Self-Supervised Learning for Text Recognition: A Critical Survey
• Predictability and Causality in Spanish and English Natural Language Generation
• UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models
• Scalable and Resource-Efficient Second-Order Federated Learning via Over-the-Air Aggregation
• Segment-Level Vectorized Beam Search Based on Partially Autoregressive Inference
• Alternate Intermediate Conditioning with Syllable-level and Character-level Targets for Japanese ASR
• An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition
• Optimal word order for non-causal text generation with Large Language Models: the Spanish case