Exploring Transformers for Large-Scale Speech Recognition

Abstract

While recurrent neural networks still largely define state-of-the-art speech recognition systems, the Transformer network has been proven to be a competitive alternative, especially in the offline condition. Most studies with Transformers have been constrained in a relatively small scale setting, and some forms of data argumentation approaches are usually applied to combat the data sparsity issue. In this paper, we aim at understanding the behaviors of Transformers in the large-scale speech recognition setting, where we have used around 65,000 hours of training data. We investigated various aspects on scaling up Transformers, including model initialization, warmup training as well as different Layer Normalization strategies. In the streaming condition, we compared the widely used attention mask based future context lookahead approach to the Transformer-XL network. From our experiments, we show that Transformers can achieve around 6% relative word error rate (WER) reduction compared to the BLSTM baseline in the offline fashion, while in the streaming fashion, Transformer-XL is comparable to LC-BLSTM with 800 millisecond latency constraint.

5 pages, 1 figure, Interspeech 2020 Camera Ready

References in corpus (4)

• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Self-Attention Transducers for End-to-End Speech Recognition
• Compressive Transformers for Long-Range Sequence Modelling
• A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models

Cited by in corpus (5)

• On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformers
• End-to-End Multi-Channel Transformer for Speech Recognition
• Transformer Based Deliberation for Two-Pass Speech Recognition
• CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognition
• Multi-Channel Transformer Transducer for Speech Recognition