SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training

Abstract

Unsupervised pre-training is now the predominant approach for both text and speech understanding. Self-attention models pre-trained on large amounts of unannotated data have been hugely successful when fine-tuned on downstream tasks from a variety of domains and languages. This paper takes the universality of unsupervised language pre-training one step further, by unifying speech and text pre-training within a single model. We build a single encoder with the BERT objective on unlabeled text together with the w2v-BERT objective on unlabeled speech. To further align our model representations across modalities, we leverage alignment losses, specifically Translation Language Modeling (TLM) and Speech Text Matching (STM) that make use of supervised speech-text recognition data. We demonstrate that incorporating both speech and text data during pre-training can significantly improve downstream quality on CoVoST~2 speech translation, by around 1 BLEU compared to single-modality pre-trained models, while retaining close to SotA performance on LibriSpeech and SpeechStew ASR tasks. On four GLUE tasks and text-normalization, we observe evidence of capacity limitations and interference between the two modalities, leading to degraded performance compared to an equivalent text-only model, while still being competitive with BERT. Through extensive empirical analysis we also demonstrate the importance of the choice of objective function for speech pre-training, and the beneficial effect of adding additional supervised signals on the quality of the learned representations.

References in corpus (20)

• Distributed Representations of Words and Phrases and their Compositionality
• Categorical Reparameterization with Gumbel-Softmax
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Common Voice: A Massively-Multilingual Speech Corpus
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
• RNN Approaches to Text Normalization: A Challenge
• CoVoST 2 and Massively Multilingual Speech-to-Text Translation
• Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder
• Unsupervised Speech Recognition
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation

Cited by in corpus (4)

• Self-Supervised Speech Representation Learning: A Review
• BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations
• VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
• Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation