Improving RNN transducer with normalized jointer network

Abstract

Recurrent neural transducer (RNN-T) is a promising end-to-end (E2E) model in automatic speech recognition (ASR). It has shown superior performance compared to traditional hybrid ASR systems. However, training RNN-T from scratch is still challenging. We observe a huge gradient variance during RNN-T training and suspect it hurts the performance. In this work, we analyze the cause of the huge gradient variance in RNN-T training and proposed a new \textit{normalized jointer network} to overcome it. We also propose to enhance the RNN-T network with a modified conformer encoder network and transformer-XL predictor networks to achieve the best performance. Experiments are conducted on the open 170-hour AISHELL-1 and industrial-level 30000-hour mandarin speech dataset. On the AISHELL-1 dataset, our RNN-T system gets state-of-the-art results on AISHELL-1's streaming and non-streaming benchmark with CER 6.15\% and 5.37\% respectively. We further compare our RNN-T system with our well trained commercial hybrid system on 30000-hour-industry audio data and get 9\% relative improvement without pre-training or external language model.

References in corpus (6)

• Sequence Transduction with Recurrent Neural Networks
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Self-Attention Transducers for End-to-End Speech Recognition
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition
• SAN-M: Memory Equipped Self-Attention for End-to-End Speech Recognition