Improving Generalization of Transformer for Speech Recognition with Parallel Schedule Sampling and Relative Positional Embedding

Abstract

Transformer has shown promising results in many sequence to sequence transformation tasks recently. It utilizes a number of feed-forward self-attention layers to replace the recurrent neural networks (RNN) in attention-based encoder decoder (AED) architecture. Self-attention layer learns temporal dependence by incorporating sinusoidal positional embedding of tokens in a sequence for parallel computing. Quicker iteration speed in training than sequential operation of RNN can be obtained. Deeper layers of the transformer also make it perform better than RNN-based AED. However, this parallelization ability is lost when applying scheduled sampling training. Self-attention with sinusoidal positional embedding may cause performance degradations for longer sequences that have similar acoustic or semantic information at different positions as well. To address these problems, we propose to use parallel scheduled sampling (PSS) and relative positional embedding (RPE) to help the transformer generalize to unseen data. Our proposed methods achieve a 7% relative improvement for short utterances and a 70% relative gain for long utterances on a 10,000-hour Mandarin ASR task.

References in corpus (8)

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Attention-Based Models for Speech Recognition
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Sequence Transduction with Recurrent Neural Networks
• A Comparative Study on Transformer vs RNN in Speech Applications
• Generating Long Sequences with Sparse Transformers
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• Parallel Scheduled Sampling

Cited by in corpus (10)

• A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition
• A Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech Recognition
• Relative Positional Encoding for Transformers with Linear Complexity
• Mitigation of Spatial Nonstationarity with Vision Transformers
• Self-and-Mixed Attention Decoder with Deep Acoustic Structure for Transformer-based LVCSR
• Transformer-based end-to-end speech recognition with residual Gaussian-based self-attention
• Reducing Exposure Bias in Training Recurrent Neural Network Transducers
• CASS-NAT: CTC Alignment-based Single Step Non-autoregressive Transformer for Speech Recognition
• Fast offline Transformer-based end-to-end automatic speech recognition for real-world applications
• An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition