Injecting Text in Self-Supervised Speech Pretraining
arXiv:2108.12226
Abstract
Self-supervised pretraining for Automated Speech Recognition (ASR) has shown varied degrees of success. In this paper, we propose to jointly learn representations during pretraining from two different modalities: speech and text. The proposed method, tts4pretrain complements the power of contrastive learning in self-supervision with linguistic/lexical representations derived from synthesized speech, effectively learning from untranscribed speech and unspoken text. Lexical learning in the speech encoder is enforced through an additional sequence loss term that is coupled with contrastive loss during pretraining. We demonstrate that this novel pretraining method yields Word Error Rate (WER) reductions of 10% relative on the well-benchmarked, Librispeech task over a state-of-the-art baseline pretrained with wav2vec2.0 only. The proposed method also serves as an effective strategy to compensate for the lack of transcribed speech, effectively matching the performance of 5000 hours of transcribed speech with just 100 hours of transcribed speech on the AMI meeting transcription task. Finally, we demonstrate WER reductions of up to 15% on an in-house Voice Search task over traditional pretraining. Incorporating text into encoder pretraining is complimentary to rescoring with a larger or in-domain language model, resulting in additional 6% relative reduction in WER.
arXiv commentsubmit to ASRU 2021
References in corpus (15)
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Sequence Transduction with Recurrent Neural Networks
- Dual Learning for Machine Translation
- On Using Monolingual Corpora in Neural Machine Translation
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Improved Noisy Student Training for Automatic Speech Recognition
- Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
- SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
- Hierarchical Generative Modeling for Controllable Speech Synthesis
- Training Neural Speech Recognition Systems with Synthetic Speech Augmentation
- Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation
- Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
- Machine Speech Chain with One-shot Speaker Adaptation
- Semi-Supervised Learning with Data Augmentation for End-to-End ASR