Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss

Abstract

In this paper we present an end-to-end speech recognition model with Transformer encoders that can be used in a streaming speech recognition system. Transformer computation blocks based on self-attention are used to encode both audio and label sequences independently. The activations from both audio and label encoders are combined with a feed-forward layer to compute a probability distribution over the label space for every combination of acoustic frame position and label history. This is similar to the Recurrent Neural Network Transducer (RNN-T) model, which uses RNNs for information encoding instead of Transformer encoders. The model is trained with the RNN-T loss well-suited to streaming decoding. We present results on the LibriSpeech dataset showing that limiting the left context for self-attention in the Transformer layers makes decoding computationally tractable for streaming, with only a slight degradation in accuracy. We also show that the full attention version of our model beats the-state-of-the art accuracy on the LibriSpeech benchmarks. Our results also show that we can bridge the gap between full attention and limited attention versions of our model by attending to a limited number of future frames.

This is the final version of the paper submitted to the ICASSP 2020 on Oct 21, 2019

References in corpus (8)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Language Modeling with Deep Transformers
• Transformers with convolutional context for ASR
• Learning Deep Transformer Models for Machine Translation
• A Neural Transducer
• Towards Online End-to-end Transformer Automatic Speech Recognition

Cited by in corpus (8)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• End-to-end Neural Diarization: From Transformer to Conformer
• Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition
• Reducing Streaming ASR Model Delay with Self Alignment
• Efficient End-to-End Speech Recognition Using Performers in Conformers
• Robustness of end-to-end Automatic Speech Recognition Models -- A Case Study using Mozilla DeepSpeech
• Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers
• On Addressing Practical Challenges for RNN-Transducer