An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition

Abstract

Non-autoregressive mechanisms can significantly decrease inference time for speech transformers, especially when the single step variant is applied. Previous work on CTC alignment-based single step non-autoregressive transformer (CASS-NAT) has shown a large real time factor (RTF) improvement over autoregressive transformers (AT). In this work, we propose several methods to improve the accuracy of the end-to-end CASS-NAT, followed by performance analyses. First, convolution augmented self-attention blocks are applied to both the encoder and decoder modules. Second, we propose to expand the trigger mask (acoustic boundary) for each token to increase the robustness of CTC alignments. In addition, iterated loss functions are used to enhance the gradient update of low-layer parameters. Without using an external language model, the WERs of the improved CASS-NAT, when using the three methods, are 3.1%/7.2% on Librispeech test clean/other sets and the CER is 5.4% on the Aishell1 test set, achieving a 7%~21% relative WER/CER improvement. For the analyses, we plot attention weight distributions in the decoders to visualize the relationships between token-level acoustic embeddings. When the acoustic embeddings are visualized, we find that they have a similar behavior to word embeddings, which explains why the improved CASS-NAT performs similarly to AT.

arXiv commentAccepted to Interspeech2021

References in corpus (8)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognition
• Non-Autoregressive Neural Text-to-Speech
• Improving Generalization of Transformer for Speech Recognition with Parallel Schedule Sampling and Relative Positional Embedding
• Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
• Intermediate Loss Regularization for CTC-based Speech Recognition