Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languages

Abstract

Sequence-to-sequence attention-based models integrate an acoustic, pronunciation and language model into a single neural network, which make them very suitable for multilingual automatic speech recognition (ASR). In this paper, we are concerned with multilingual speech recognition on low-resource languages by a single Transformer, one of sequence-to-sequence attention-based models. Sub-words are employed as the multilingual modeling unit without using any pronunciation lexicon. First, we show that a single multilingual ASR Transformer performs well on low-resource languages despite of some language confusion. We then look at incorporating language information into the model by inserting the language symbol at the beginning or at the end of the original sub-words sequence under the condition of language information being known during training. Experiments on CALLHOME datasets demonstrate that the multilingual ASR Transformer with the language symbol at the end performs better and can obtain relatively 10.5\% average word error rate (WER) reduction compared to SHL-MLSTM with residual learning. We go on to show that, assuming the language information being known during training and testing, about relatively 12.4\% average WER reduction can be observed compared to SHL-MLSTM with residual learning through giving the language symbol as the sentence start token.

arXiv commentarXiv admin note: text overlap with arXiv:1805.06239

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese
• Multilingual Speech Recognition With A Single End-To-End Model
• A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese
• Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model
• Towards Language-Universal End-to-End Speech Recognition

Cited by in corpus (14)

• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages
• Improved Meta Learning for Low Resource Speech Recognition
• Scaling End-to-End Models for Large-Scale Multilingual ASR
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
• OLR 2021 Challenge: Datasets, Rules and Baselines
• Masked Pre-trained Encoder base on Joint CTC-Transformer
• Synchronous Bidirectional Learning for Multilingual Lip Reading
• Transfer Learning for Multi-lingual Tasks -- a Survey
• Multilingual Speech Recognition for Low-Resource Indian Languages using Multi-Task conformer
• Flexi-Transducer: Optimizing Latency, Accuracy and Compute forMulti-Domain On-Device Scenarios
• A Dual-Decoder Conformer for Multilingual Speech Recognition
• Fast offline Transformer-based end-to-end automatic speech recognition for real-world applications
• Integrating Categorical Features in End-to-End ASR