SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network

Abstract

We present SpeechStew, a speech recognition model that is trained on a combination of various publicly available speech recognition datasets: AMI, Broadcast News, Common Voice, LibriSpeech, Switchboard/Fisher, Tedlium, and Wall Street Journal. SpeechStew simply mixes all of these datasets together, without any special re-weighting or re-balancing of the datasets. SpeechStew achieves SoTA or near SoTA results across a variety of tasks, without the use of an external language model. Our results include 9.0\% WER on AMI-IHM, 4.7\% WER on Switchboard, 8.3\% WER on CallHome, and 1.3\% on WSJ, which significantly outperforms prior work with strong external language models. We also demonstrate that SpeechStew learns powerful transfer learning representations. We fine-tune SpeechStew on a noisy low resource speech dataset, CHiME-6. We achieve 38.9\% WER without a language model, which compares to 38.6\% WER to a strong HMM baseline with a language model.

arXiv commentsubmitted to INTERSPEECH

References in corpus (13)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Scaling Laws for Neural Language Models
• Sequence Transduction with Recurrent Neural Networks
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation
• Improved Noisy Student Training for Automatic Speech Recognition
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings
• Towards a Competitive End-to-End Speech Recognition for CHiME-6 Dinner Party Transcription
• Transformer Language Models with LSTM-based Cross-utterance Information Representation
• Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone

Cited by in corpus (14)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
• Measuring the Accuracy of Automatic Speech Recognition Solutions
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
• A deep convolutional neural network that is invariant to time rescaling
• Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development
• Injecting Text in Self-Supervised Speech Pretraining
• Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder
• Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone
• DeCoR: Defy Knowledge Forgetting by Predicting Earlier Audio Codes
• CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription