Semi-Supervised Speech Recognition via Local Prior Matching

Abstract

For sequence transduction tasks like speech recognition, a strong structured prior model encodes rich information about the target space, implicitly ruling out invalid sequences by assigning them low probability. In this work, we propose local prior matching (LPM), a semi-supervised objective that distills knowledge from a strong prior (e.g. a language model) to provide learning signal to a discriminative model trained on unlabeled speech. We demonstrate that LPM is theoretically well-motivated, simple to implement, and superior to existing knowledge distillation techniques under comparable settings. Starting from a baseline trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data, LPM recovers 54% and 73% of the word error rate on clean and noisy test sets relative to a fully supervised model on the same data.

References in corpus (12)

• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• wav2letter++: The Fastest Open-source Speech Recognition System
• Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation
• Fully Convolutional Speech Recognition
• Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
• Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundaries
• Listening while Speaking: Speech Chain by Deep Learning
• A Fully Differentiable Beam Search Decoder
• Attentive Adversarial Learning for Domain-Invariant Training
• Back-Translation-Style Data Augmentation for End-to-End ASR

Cited by in corpus (10)

• Improved Noisy Student Training for Automatic Speech Recognition
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Iterative Pseudo-Labeling for Speech Recognition
• Unsupervised Speech Recognition
• Self-training and Pre-training are Complementary for Speech Recognition
• Towards Semi-Supervised Semantics Understanding from Speech
• Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
• Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation
• A Unified Speaker Adaptation Approach for ASR