Iterative Pseudo-Labeling for Speech Recognition

Abstract

Pseudo-labeling has recently shown promise in end-to-end automatic speech recognition (ASR). We study Iterative Pseudo-Labeling (IPL), a semi-supervised algorithm which efficiently performs multiple iterations of pseudo-labeling on unlabeled data as the acoustic model evolves. In particular, IPL fine-tunes an existing model at each iteration using both labeled data and a subset of unlabeled data. We study the main components of IPL: decoding with a language model and data augmentation. We then demonstrate the effectiveness of IPL by achieving state-of-the-art word-error rate on the Librispeech test sets in both standard and low-resource setting. We also study the effect of language models trained on different corpora to show IPL can effectively utilize additional text. Finally, we release a new large in-domain text corpus which does not overlap with the Librispeech training transcriptions to foster research in low-resource, semi-supervised ASR

INTERSPEECH 2020

References in corpus (14)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• A Comparative Study on Transformer vs RNN in Speech Applications
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Improved Noisy Student Training for Automatic Speech Recognition
• Self-Training for End-to-End Speech Recognition
• wav2letter++: The Fastest Open-source Speech Recognition System
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Transformers with convolutional context for ASR
• Effectiveness of self-supervised pre-training for speech recognition
• Fully Convolutional Speech Recognition
• Semi-Supervised Speech Recognition via Local Prior Matching
• Semi-supervised ASR by End-to-end Self-training

Cited by in corpus (9)

• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• The History of Speech Recognition to the Year 2030
• Self-training and Pre-training are Complementary for Speech Recognition
• Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0
• Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
• Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
• Semi-supervised ASR by End-to-end Self-training
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• Towards Semi-Supervised Semantics Understanding from Speech