Semantic Mask for Transformer based End-to-End Speech Recognition

Abstract

Attention-based encoder-decoder model has achieved impressive results for both automatic speech recognition (ASR) and text-to-speech (TTS) tasks. This approach takes advantage of the memorization capacity of neural networks to learn the mapping from the input sequence to the output sequence from scratch, without the assumption of prior knowledge such as the alignments. However, this model is prone to overfitting, especially when the amount of training data is limited. Inspired by SpecAugment and BERT, in this paper, we propose a semantic mask based regularization for training such kind of end-to-end (E2E) model. The idea is to mask the input features corresponding to a particular output token, e.g., a word or a word-piece, in order to encourage the model to fill the token based on the contextual information. While this approach is applicable to the encoder-decoder framework with any type of neural network architecture, we study the transformer-based model for ASR in this work. We perform experiments on Librispeech 960h and TedLium2 data sets, and achieve the state-of-the-art performance on the test set in the scope of E2E models.

References in corpus (7)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• A Comparative Study on Transformer vs RNN in Speech Applications
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• Transformers with convolutional context for ASR
• SpecAugment on Large Scale Datasets

Cited by in corpus (13)

• Online Hybrid CTC/Attention End-to-End Automatic Speech Recognition Architecture
• Low Latency End-to-End Streaming Speech Recognition with a Scout Network
• Ultra Fast Speech Separation Model with Teacher Student Learning
• Exploring Transformers for Large-Scale Speech Recognition
• Curriculum Pre-training for End-to-End Speech Translation
• Leveraging Phone Mask Training for Phonetic-Reduction-Robust E2E Uyghur Speech Recognition
• MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognition
• Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers
• Don't shoot butterfly with rifles: Multi-channel Continuous Speech Separation with Early Exit Transformer
• High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model
• Effective Decoder Masking for Transformer Based End-to-End Speech Recognition
• Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation
• Investigation of Practical Aspects of Single Channel Speech Separation for ASR