ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding

Abstract

Language model pre-training has shown promising results in various downstream tasks. In this context, we introduce a cross-modal pre-trained language model, called Speech-Text BERT (ST-BERT), to tackle end-to-end spoken language understanding (E2E SLU) tasks. Taking phoneme posterior and subword-level text as an input, ST-BERT learns a contextualized cross-modal alignment via our two proposed pre-training tasks: Cross-modal Masked Language Modeling (CM-MLM) and Cross-modal Conditioned Language Modeling (CM-CLM). Experimental results on three benchmarks present that our approach is effective for various SLU datasets and shows a surprisingly marginal performance degradation even when 1% of the training data are available. Also, our method shows further SLU performance gain via domain-adaptive pre-training with domain-specific speech-text pair data.

arXiv commentICASSP 2021; 5 pages

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Cross-lingual Language Model Pretraining
• Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• NSML: Meet the MLaaS platform with a real-world case study
• Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering