Improved Noisy Student Training for Automatic Speech Recognition

Abstract

Recently, a semi-supervised learning method known as "noisy student training" has been shown to improve image classification performance of deep networks significantly. Noisy student training is an iterative self-training method that leverages augmentation to improve network performance. In this work, we adapt and improve noisy student training for automatic speech recognition, employing (adaptive) SpecAugment as the augmentation method. We find effective methods to filter, balance and augment the data generated in between self-training iterations. By doing so, we are able to obtain word error rates (WERs) 4.2%/8.6% on the clean/noisy LibriSpeech test sets by only using the clean 100h subset of LibriSpeech as the supervised set and the rest (860h) as the unlabeled set. Furthermore, we are able to achieve WERs 1.7%/3.4% on the clean/noisy LibriSpeech test sets by using the unlab-60k subset of LibriLight as the unlabeled set for LibriSpeech 960h. We are thus able to improve upon the previous state-of-the-art clean/noisy test WERs achieved on LibriSpeech 100h (4.74%/12.20%) and LibriSpeech (1.9%/4.1%).

arXiv comment5 pages, 5 figures, 4 tables; v2: minor revisions, reference added

References in corpus (12)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Sequence Transduction with Recurrent Neural Networks
• On Using Monolingual Corpora in Neural Machine Translation
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Self-Training for End-to-End Speech Recognition
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation
• Revisiting Self-Training for Neural Sequence Generation
• Semi-Supervised Speech Recognition via Local Prior Matching

Cited by in corpus (49)

• Self-Supervised Speech Representation Learning: A Review
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Self-training Improves Pre-training for Natural Language Understanding
• Iterative Pseudo-Labeling for Speech Recognition
• Poisoning the Unlabeled Dataset of Semi-Supervised Learning
• Unsupervised Speech Recognition
• PseudoAugment: Learning to Use Unlabeled Data for Data Augmentation in Point Clouds
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Self-training and Pre-training are Complementary for Speech Recognition
• KeepAugment: A Simple Information-Preserving Data Augmentation Approach
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• Swiss Parliaments Corpus, an Automatically Aligned Swiss German Speech to Standard German Text Corpus
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• Semi-supervised ASR by End-to-end Self-training
• Textual Echo Cancellation
• Towards Semi-Supervised Semantics Understanding from Speech
• Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation
• Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensors
• End-to-end Jordanian dialect speech-to-text self-supervised learning framework
• Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
• Pseudo-labeling for Scalable 3D Object Detection
• Injecting Text in Self-Supervised Speech Pretraining
• Overcoming Domain Mismatch in Low Resource Sequence-to-Sequence ASR Models using Hybrid Generated Pseudotranscripts
• Residual Energy-Based Models for End-to-End Speech Recognition
• Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition
• Learning Word-Level Confidence For Subword End-to-End ASR
• Semi-supervised transfer learning for language expansion of end-to-end speech recognition models to low-resource languages
• Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
• End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend
• Contrastive Semi-supervised Learning for ASR
• Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces
• Semi-Supervised Singing Voice Separation with Noisy Self-Training
• Training Aware Sigmoidal Optimizer
• Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation
• Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data
• Noised Consistency Training for Text Summarization
• ASR Rescoring and Confidence Estimation with ELECTRA
• Topic Model Robustness to Automatic Speech Recognition Errors in Podcast Transcripts
• Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems