XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition

Abstract

In this paper, we propose a weakly supervised multilingual representation learning framework, called cross-lingual self-training (XLST). XLST is able to utilize a small amount of annotated data from high-resource languages to improve the representation learning on multilingual un-annotated data. Specifically, XLST uses a supervised trained model to produce initial representations and another model to learn from them, by maximizing the similarity between output embeddings of these two models. Furthermore, the moving average mechanism and multi-view data augmentation are employed, which are experimentally shown to be crucial to XLST. Comprehensive experiments have been conducted on the CommonVoice corpus to evaluate the effectiveness of XLST. Results on 5 downstream low-resource ASR tasks shows that our multilingual pretrained model achieves relatively 18.6% PER reduction over the state-of-the-art self-supervised method, with leveraging additional 100 hours of annotated English data.

arXiv comment5 pages, 1 figure

References in corpus (12)

• mixup: Beyond Empirical Risk Minimization
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Interpolation Consistency Training for Semi-Supervised Learning
• Exploring Simple Siamese Representation Learning
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Improved Noisy Student Training for Automatic Speech Recognition
• Self-Training for End-to-End Speech Recognition
• Transformers with convolutional context for ASR
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data