Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition

Abstract

We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.

arXiv comment11 pages, 3 figures, 5 tables. Accepted to NeurIPS SAS 2020 Workshop; v2: minor errors corrected

References in corpus (24)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Unsupervised Data Augmentation for Consistency Training
• Sequence Transduction with Recurrent Neural Networks
• Searching for Activation Functions
• On Using Monolingual Corpora in Neural Machine Translation
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Rethinking Pre-training and Self-training
• Conformer: Convolution-augmented Transformer for Speech Recognition
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Improved Noisy Student Training for Automatic Speech Recognition
• Self-Training for End-to-End Speech Recognition
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Revisiting Self-Training for Neural Sequence Generation
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• An Unsupervised Autoregressive Model for Speech Representation Learning
• Jasper: An End-to-End Convolutional Neural Acoustic Model
• Letter-Based Speech Recognition with Gated ConvNets
• Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
• Semi-Supervised Speech Recognition via Local Prior Matching
• 0-1 phase transitions in sparse spiked matrix estimation

Cited by in corpus (40)

• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
• Transfer Learning and SpecAugment applied to SSVEP Based BCI Classification
• Earnings-21: A Practical Benchmark for ASR in the Wild
• Universal Paralinguistic Speech Representations Using Self-Supervised Conformers
• Unsupervised Speech Recognition
• Pushing the Limits of Non-Autoregressive Speech Recognition
• The History of Speech Recognition to the Year 2030
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• MediaSpeech: Multilanguage ASR Benchmark and Dataset
• Scaling End-to-End Models for Large-Scale Multilingual ASR
• When are Deep Networks really better than Decision Forests at small sample sizes, and how?
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Data Augmentation For Children's Speech Recognition -- The "Ethiopian" System For The SLT 2021 Children Speech Recognition Challenge
• Unsupervised data selection for Speech Recognition with contrastive loss ratios
• UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
• OverFlow: Putting flows on top of neural transducers for better TTS
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
• Detection of Prosodic Boundaries in Speech Using Wav2Vec 2.0
• Contextual Semi-Supervised Learning: An Approach To Leverage Air-Surveillance and Untranscribed ATC Data in ASR Systems
• Multi-Task Self-Training for Learning General Representations
• Conditional independence for pretext task selection in Self-supervised speech representation learning
• An Investigation of the Effectiveness of Phase for Audio Classification
• Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation
• End-to-end Jordanian dialect speech-to-text self-supervised learning framework
• Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder
• Injecting Text in Self-Supervised Speech Pretraining
• Multilingual Speech Recognition using Knowledge Transfer across Learning Processes
• Residual Energy-Based Models for End-to-End Speech Recognition
• Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutch
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
• Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models
• Semi-Supervised Vision Transformers
• CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription
• Pretext Tasks selection for multitask self-supervised speech representation learning
• Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning
• Topic Model Robustness to Automatic Speech Recognition Errors in Podcast Transcripts
• Multi-mode Transformer Transducer with Stochastic Future Context