W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
arXiv:2108.06209
Abstract
Motivated by the success of masked language modeling~(MLM) in pre-training natural language processing models, we propose w2v-BERT that explores MLM for self-supervised speech representation learning. w2v-BERT is a framework that combines contrastive learning and MLM, where the former trains the model to discretize input continuous speech signals into a finite set of discriminative speech tokens, and the latter trains the model to learn contextualized speech representations via solving a masked prediction task consuming the discretized tokens. In contrast to existing MLM-based speech pre-training frameworks such as HuBERT, which relies on an iterative re-clustering and re-training process, or vq-wav2vec, which concatenates two separately trained modules, w2v-BERT can be optimized in an end-to-end fashion by solving the two self-supervised tasks~(the contrastive task and MLM) simultaneously. Our experiments show that w2v-BERT achieves competitive results compared to current state-of-the-art pre-trained models on the LibriSpeech benchmarks when using the Libri-Light~60k corpus as the unsupervised data. In particular, when compared to published models such as conformer-based wav2vec~2.0 and HuBERT, our model shows~5\% to~10\% relative WER reduction on the test-clean and test-other subsets. When applied to the Google's Voice Search traffic dataset, w2v-BERT outperforms our internal conformer-based wav2vec~2.0 by more than~30\% relatively.
References in corpus (15)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Sequence Transduction with Recurrent Neural Networks
- Searching for Activation Functions
- Libri-Light: A Benchmark for ASR with Limited or No Supervision
- Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
- TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
- Improved Noisy Student Training for Automatic Speech Recognition
- Self-Training for End-to-End Speech Recognition
- Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
- End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
- Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
- Effectiveness of self-supervised pre-training for speech recognition
- DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
- Scaling End-to-End Models for Large-Scale Multilingual ASR