Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
arXiv:2106.08922
Abstract
Pseudo-labeling (PL) has been shown to be effective in semi-supervised automatic speech recognition (ASR), where a base model is self-trained with pseudo-labels generated from unlabeled data. While PL can be further improved by iteratively updating pseudo-labels as the model evolves, most of the previous approaches involve inefficient retraining of the model or intricate control of the label update. We present momentum pseudo-labeling (MPL), a simple yet effective strategy for semi-supervised ASR. MPL consists of a pair of online and offline models that interact and learn from each other, inspired by the mean teacher method. The online model is trained to predict pseudo-labels generated on the fly by the offline model. The offline model maintains a momentum-based moving average of the online model. MPL is performed in a single training process and the interaction between the two models effectively helps them reinforce each other to improve the ASR performance. We apply MPL to an end-to-end ASR model based on the connectionist temporal classification. The experimental results demonstrate that MPL effectively improves over the base model and is scalable to different semi-supervised scenarios with varying amounts of data or domain mismatch.
arXiv commentAccepted to Interspeech 2021
References in corpus (10)
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Attention-Based Models for Speech Recognition
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- A Comparative Study on Transformer vs RNN in Speech Applications
- Improved Noisy Student Training for Automatic Speech Recognition
- Self-Training for End-to-End Speech Recognition
- Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
- Effectiveness of self-supervised pre-training for speech recognition
- Almost Unsupervised Text to Speech and Automatic Speech Recognition
- Semi-Supervised Speech Recognition via Local Prior Matching